<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.11">
<meta name="description" content="The perfect emulation setup to study and develop the &lt;&lt;linux-kernel&gt;&gt; v5.9.2, kernel modules, &lt;&lt;qemu-buildroot-setup,QEMU&gt;&gt;, &lt;&lt;gem5-buildroot-setup,gem5&gt;&gt; and x86_64, ARMv7 and ARMv8 &lt;&lt;userland-assembly,userland&gt;&gt; and &lt;&lt;baremetal-setup,baremetal&gt;&gt; assembly, &lt;&lt;c,ANSI C&gt;&gt;, &lt;&lt;cpp,C++&gt;&gt; and &lt;&lt;posix,POSIX&gt;&gt;. &lt;&lt;gdb&gt;&gt; and &lt;&lt;kgdb&gt;&gt; just work. Powered by &lt;&lt;about-the-qemu-buildroot-setup,Buildroot&gt;&gt; and &lt;&lt;about-the-baremetal-setup,crosstool-NG&gt;&gt;. Highly automated. Thoroughly documented. Automated &lt;&lt;test-this-repo,tests&gt;&gt;. "Tested" in an Ubuntu 20.04 host.">
<title>Linux Kernel Module Cheat</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
/* Asciidoctor default stylesheet | MIT License | https://asciidoctor.org */
/* Uncomment @import statement to use as custom stylesheet */
/*@import "https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700";*/
article,aside,details,figcaption,figure,footer,header,hgroup,main,nav,section{display:block}
audio,video{display:inline-block}
audio:not([controls]){display:none;height:0}
html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}
a{background:none}
a:focus{outline:thin dotted}
a:active,a:hover{outline:0}
h1{font-size:2em;margin:.67em 0}
abbr[title]{border-bottom:1px dotted}
b,strong{font-weight:bold}
dfn{font-style:italic}
hr{-moz-box-sizing:content-box;box-sizing:content-box;height:0}
mark{background:#ff0;color:#000}
code,kbd,pre,samp{font-family:monospace;font-size:1em}
pre{white-space:pre-wrap}
q{quotes:"\201C" "\201D" "\2018" "\2019"}
small{font-size:80%}
sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}
sup{top:-.5em}
sub{bottom:-.25em}
img{border:0}
svg:not(:root){overflow:hidden}
figure{margin:0}
fieldset{border:1px solid silver;margin:0 2px;padding:.35em .625em .75em}
legend{border:0;padding:0}
button,input,select,textarea{font-family:inherit;font-size:100%;margin:0}
button,input{line-height:normal}
button,select{text-transform:none}
button,html input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer}
button[disabled],html input[disabled]{cursor:default}
input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0}
button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}
textarea{overflow:auto;vertical-align:top}
table{border-collapse:collapse;border-spacing:0}
*,*::before,*::after{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box}
html,body{font-size:100%}
body{background:#fff;color:rgba(0,0,0,.8);padding:0;margin:0;font-family:"Noto Serif","DejaVu Serif",serif;font-weight:400;font-style:normal;line-height:1;position:relative;cursor:auto;tab-size:4;word-wrap:anywhere;-moz-osx-font-smoothing:grayscale;-webkit-font-smoothing:antialiased}
a:hover{cursor:pointer}
img,object,embed{max-width:100%;height:auto}
object,embed{height:100%}
img{-ms-interpolation-mode:bicubic}
.left{float:left!important}
.right{float:right!important}
.text-left{text-align:left!important}
.text-right{text-align:right!important}
.text-center{text-align:center!important}
.text-justify{text-align:justify!important}
.hide{display:none}
img,object,svg{display:inline-block;vertical-align:middle}
textarea{height:auto;min-height:50px}
select{width:100%}
.subheader,.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{line-height:1.45;color:#7a2518;font-weight:400;margin-top:0;margin-bottom:.25em}
div,dl,dt,dd,ul,ol,li,h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0}
a{color:#2156a5;text-decoration:underline;line-height:inherit}
a:hover,a:focus{color:#1d4b8f}
a img{border:0}
p{font-family:inherit;font-weight:400;font-size:1em;line-height:1.6;margin-bottom:1.25em;text-rendering:optimizeLegibility}
p aside{font-size:.875em;line-height:1.35;font-style:italic}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{font-family:"Open Sans","DejaVu Sans",sans-serif;font-weight:300;font-style:normal;color:#ba3925;text-rendering:optimizeLegibility;margin-top:1em;margin-bottom:.5em;line-height:1.0125em}
h1 small,h2 small,h3 small,#toctitle small,.sidebarblock>.content>.title small,h4 small,h5 small,h6 small{font-size:60%;color:#e99b8f;line-height:0}
h1{font-size:2.125em}
h2{font-size:1.6875em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.375em}
h4,h5{font-size:1.125em}
h6{font-size:1em}
hr{border:solid #dddddf;border-width:1px 0 0;clear:both;margin:1.25em 0 1.1875em;height:0}
em,i{font-style:italic;line-height:inherit}
strong,b{font-weight:bold;line-height:inherit}
small{font-size:60%;line-height:inherit}
code{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;font-weight:400;color:rgba(0,0,0,.9)}
ul,ol,dl{font-size:1em;line-height:1.6;margin-bottom:1.25em;list-style-position:outside;font-family:inherit}
ul,ol{margin-left:1.5em}
ul li ul,ul li ol{margin-left:1.25em;margin-bottom:0;font-size:1em}
ul.square li ul,ul.circle li ul,ul.disc li ul{list-style:inherit}
ul.square{list-style-type:square}
ul.circle{list-style-type:circle}
ul.disc{list-style-type:disc}
ol li ul,ol li ol{margin-left:1.25em;margin-bottom:0}
dl dt{margin-bottom:.3125em;font-weight:bold}
dl dd{margin-bottom:1.25em}
abbr,acronym{text-transform:uppercase;font-size:90%;color:rgba(0,0,0,.8);border-bottom:1px dotted #ddd;cursor:help}
abbr{text-transform:none}
blockquote{margin:0 0 1.25em;padding:.5625em 1.25em 0 1.1875em;border-left:1px solid #ddd}
blockquote cite{display:block;font-size:.9375em;color:rgba(0,0,0,.6)}
blockquote cite::before{content:"\2014 \0020"}
blockquote cite a,blockquote cite a:visited{color:rgba(0,0,0,.6)}
blockquote,blockquote p{line-height:1.6;color:rgba(0,0,0,.85)}
@media screen and (min-width:768px){h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2}
h1{font-size:2.75em}
h2{font-size:2.3125em}
h3,#toctitle,.sidebarblock>.content>.title{font-size:1.6875em}
h4{font-size:1.4375em}}
table{background:#fff;margin-bottom:1.25em;border:solid 1px #dedede}
table thead,table tfoot{background:#f7f8f7}
table thead tr th,table thead tr td,table tfoot tr th,table tfoot tr td{padding:.5em .625em .625em;font-size:inherit;color:rgba(0,0,0,.8);text-align:left}
table tr th,table tr td{padding:.5625em .625em;font-size:inherit;color:rgba(0,0,0,.8)}
table tr.even,table tr.alt{background:#f8f8f7}
table thead tr th,table tfoot tr th,table tbody tr td,table tr td,table tfoot tr td{display:table-cell;line-height:1.6}
h1,h2,h3,#toctitle,.sidebarblock>.content>.title,h4,h5,h6{line-height:1.2;word-spacing:-.05em}
h1 strong,h2 strong,h3 strong,#toctitle strong,.sidebarblock>.content>.title strong,h4 strong,h5 strong,h6 strong{font-weight:400}
.center{margin-left:auto;margin-right:auto}
.stretch{width:100%}
.clearfix::before,.clearfix::after,.float-group::before,.float-group::after{content:" ";display:table}
.clearfix::after,.float-group::after{clear:both}
:not(pre).nobreak{word-wrap:normal}
:not(pre).nowrap{white-space:nowrap}
:not(pre).pre-wrap{white-space:pre-wrap}
:not(pre):not([class^=L])>code{font-size:.9375em;font-style:normal!important;letter-spacing:0;padding:.1em .5ex;word-spacing:-.15em;background:#f7f7f8;-webkit-border-radius:4px;border-radius:4px;line-height:1.45;text-rendering:optimizeSpeed}
pre{color:rgba(0,0,0,.9);font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;line-height:1.45;text-rendering:optimizeSpeed}
pre code,pre pre{color:inherit;font-size:inherit;line-height:inherit}
pre>code{display:block}
pre.nowrap,pre.nowrap pre{white-space:pre;word-wrap:normal}
em em{font-style:normal}
strong strong{font-weight:400}
.keyseq{color:rgba(51,51,51,.8)}
kbd{font-family:"Droid Sans Mono","DejaVu Sans Mono",monospace;display:inline-block;color:rgba(0,0,0,.8);font-size:.65em;line-height:1.45;background:#f7f7f7;border:1px solid #ccc;-webkit-border-radius:3px;border-radius:3px;-webkit-box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em white inset;box-shadow:0 1px 0 rgba(0,0,0,.2),0 0 0 .1em #fff inset;margin:0 .15em;padding:.2em .5em;vertical-align:middle;position:relative;top:-.1em;white-space:nowrap}
.keyseq kbd:first-child{margin-left:0}
.keyseq kbd:last-child{margin-right:0}
.menuseq,.menuref{color:#000}
.menuseq b:not(.caret),.menuref{font-weight:inherit}
.menuseq{word-spacing:-.02em}
.menuseq b.caret{font-size:1.25em;line-height:.8}
.menuseq i.caret{font-weight:bold;text-align:center;width:.45em}
b.button::before,b.button::after{position:relative;top:-1px;font-weight:400}
b.button::before{content:"[";padding:0 3px 0 2px}
b.button::after{content:"]";padding:0 2px 0 3px}
p a>code:hover{color:rgba(0,0,0,.9)}
#header,#content,#footnotes,#footer{width:100%;margin-left:auto;margin-right:auto;margin-top:0;margin-bottom:0;max-width:62.5em;*zoom:1;position:relative;padding-left:.9375em;padding-right:.9375em}
#header::before,#header::after,#content::before,#content::after,#footnotes::before,#footnotes::after,#footer::before,#footer::after{content:" ";display:table}
#header::after,#content::after,#footnotes::after,#footer::after{clear:both}
#content{margin-top:1.25em}
#content::before{content:none}
#header>h1:first-child{color:rgba(0,0,0,.85);margin-top:2.25rem;margin-bottom:0}
#header>h1:first-child+#toc{margin-top:8px;border-top:1px solid #dddddf}
#header>h1:only-child,body.toc2 #header>h1:nth-last-child(2){border-bottom:1px solid #dddddf;padding-bottom:8px}
#header .details{border-bottom:1px solid #dddddf;line-height:1.45;padding-top:.25em;padding-bottom:.25em;padding-left:.25em;color:rgba(0,0,0,.6);display:-ms-flexbox;display:-webkit-flex;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap}
#header .details span:first-child{margin-left:-.125em}
#header .details span.email a{color:rgba(0,0,0,.85)}
#header .details br{display:none}
#header .details br+span::before{content:"\00a0\2013\00a0"}
#header .details br+span.author::before{content:"\00a0\22c5\00a0";color:rgba(0,0,0,.85)}
#header .details br+span#revremark::before{content:"\00a0|\00a0"}
#header #revnumber{text-transform:capitalize}
#header #revnumber::after{content:"\00a0"}
#content>h1:first-child:not([class]){color:rgba(0,0,0,.85);border-bottom:1px solid #dddddf;padding-bottom:8px;margin-top:0;padding-top:1rem;margin-bottom:1.25rem}
#toc{border-bottom:1px solid #e7e7e9;padding-bottom:.5em}
#toc>ul{margin-left:.125em}
#toc ul.sectlevel0>li>a{font-style:italic}
#toc ul.sectlevel0 ul.sectlevel1{margin:.5em 0}
#toc ul{font-family:"Open Sans","DejaVu Sans",sans-serif;list-style-type:none}
#toc li{line-height:1.3334;margin-top:.3334em}
#toc a{text-decoration:none}
#toc a:active{text-decoration:underline}
#toctitle{color:#7a2518;font-size:1.2em}
@media screen and (min-width:768px){#toctitle{font-size:1.375em}
body.toc2{padding-left:15em;padding-right:0}
#toc.toc2{margin-top:0!important;background:#f8f8f7;position:fixed;width:15em;left:0;top:0;border-right:1px solid #e7e7e9;border-top-width:0!important;border-bottom-width:0!important;z-index:1000;padding:1.25em 1em;height:100%;overflow:auto}
#toc.toc2 #toctitle{margin-top:0;margin-bottom:.8rem;font-size:1.2em}
#toc.toc2>ul{font-size:.9em;margin-bottom:0}
#toc.toc2 ul ul{margin-left:0;padding-left:1em}
#toc.toc2 ul.sectlevel0 ul.sectlevel1{padding-left:0;margin-top:.5em;margin-bottom:.5em}
body.toc2.toc-right{padding-left:0;padding-right:15em}
body.toc2.toc-right #toc.toc2{border-right-width:0;border-left:1px solid #e7e7e9;left:auto;right:0}}
@media screen and (min-width:1280px){body.toc2{padding-left:20em;padding-right:0}
#toc.toc2{width:20em}
#toc.toc2 #toctitle{font-size:1.375em}
#toc.toc2>ul{font-size:.95em}
#toc.toc2 ul ul{padding-left:1.25em}
body.toc2.toc-right{padding-left:0;padding-right:20em}}
#content #toc{border-style:solid;border-width:1px;border-color:#e0e0dc;margin-bottom:1.25em;padding:1.25em;background:#f8f8f7;-webkit-border-radius:4px;border-radius:4px}
#content #toc>:first-child{margin-top:0}
#content #toc>:last-child{margin-bottom:0}
#footer{max-width:none;background:rgba(0,0,0,.8);padding:1.25em}
#footer-text{color:rgba(255,255,255,.8);line-height:1.44}
#content{margin-bottom:.625em}
.sect1{padding-bottom:.625em}
@media screen and (min-width:768px){#content{margin-bottom:1.25em}
.sect1{padding-bottom:1.25em}}
.sect1:last-child{padding-bottom:0}
.sect1+.sect1{border-top:1px solid #e7e7e9}
#content h1>a.anchor,h2>a.anchor,h3>a.anchor,#toctitle>a.anchor,.sidebarblock>.content>.title>a.anchor,h4>a.anchor,h5>a.anchor,h6>a.anchor{position:absolute;z-index:1001;width:1.5ex;margin-left:-1.5ex;display:block;text-decoration:none!important;visibility:hidden;text-align:center;font-weight:400}
#content h1>a.anchor::before,h2>a.anchor::before,h3>a.anchor::before,#toctitle>a.anchor::before,.sidebarblock>.content>.title>a.anchor::before,h4>a.anchor::before,h5>a.anchor::before,h6>a.anchor::before{content:"\00A7";font-size:.85em;display:block;padding-top:.1em}
#content h1:hover>a.anchor,#content h1>a.anchor:hover,h2:hover>a.anchor,h2>a.anchor:hover,h3:hover>a.anchor,#toctitle:hover>a.anchor,.sidebarblock>.content>.title:hover>a.anchor,h3>a.anchor:hover,#toctitle>a.anchor:hover,.sidebarblock>.content>.title>a.anchor:hover,h4:hover>a.anchor,h4>a.anchor:hover,h5:hover>a.anchor,h5>a.anchor:hover,h6:hover>a.anchor,h6>a.anchor:hover{visibility:visible}
#content h1>a.link,h2>a.link,h3>a.link,#toctitle>a.link,.sidebarblock>.content>.title>a.link,h4>a.link,h5>a.link,h6>a.link{color:#ba3925;text-decoration:none}
#content h1>a.link:hover,h2>a.link:hover,h3>a.link:hover,#toctitle>a.link:hover,.sidebarblock>.content>.title>a.link:hover,h4>a.link:hover,h5>a.link:hover,h6>a.link:hover{color:#a53221}
details,.audioblock,.imageblock,.literalblock,.listingblock,.stemblock,.videoblock{margin-bottom:1.25em}
details>summary:first-of-type{cursor:pointer;display:list-item;outline:none;margin-bottom:.75em}
.admonitionblock td.content>.title,.audioblock>.title,.exampleblock>.title,.imageblock>.title,.listingblock>.title,.literalblock>.title,.stemblock>.title,.openblock>.title,.paragraph>.title,.quoteblock>.title,table.tableblock>.title,.verseblock>.title,.videoblock>.title,.dlist>.title,.olist>.title,.ulist>.title,.qlist>.title,.hdlist>.title{text-rendering:optimizeLegibility;text-align:left;font-family:"Noto Serif","DejaVu Serif",serif;font-size:1rem;font-style:italic}
table.tableblock.fit-content>caption.title{white-space:nowrap;width:0}
.paragraph.lead>p,#preamble>.sectionbody>[class="paragraph"]:first-of-type p{font-size:1.21875em;line-height:1.6;color:rgba(0,0,0,.85)}
table.tableblock #preamble>.sectionbody>[class="paragraph"]:first-of-type p{font-size:inherit}
.admonitionblock>table{border-collapse:separate;border:0;background:none;width:100%}
.admonitionblock>table td.icon{text-align:center;width:80px}
.admonitionblock>table td.icon img{max-width:none}
.admonitionblock>table td.icon .title{font-weight:bold;font-family:"Open Sans","DejaVu Sans",sans-serif;text-transform:uppercase}
.admonitionblock>table td.content{padding-left:1.125em;padding-right:1.25em;border-left:1px solid #dddddf;color:rgba(0,0,0,.6)}
.admonitionblock>table td.content>:last-child>:last-child{margin-bottom:0}
.exampleblock>.content{border-style:solid;border-width:1px;border-color:#e6e6e6;margin-bottom:1.25em;padding:1.25em;background:#fff;-webkit-border-radius:4px;border-radius:4px}
.exampleblock>.content>:first-child{margin-top:0}
.exampleblock>.content>:last-child{margin-bottom:0}
.sidebarblock{border-style:solid;border-width:1px;border-color:#dbdbd6;margin-bottom:1.25em;padding:1.25em;background:#f3f3f2;-webkit-border-radius:4px;border-radius:4px}
.sidebarblock>:first-child{margin-top:0}
.sidebarblock>:last-child{margin-bottom:0}
.sidebarblock>.content>.title{color:#7a2518;margin-top:0;text-align:center}
.exampleblock>.content>:last-child>:last-child,.exampleblock>.content .olist>ol>li:last-child>:last-child,.exampleblock>.content .ulist>ul>li:last-child>:last-child,.exampleblock>.content .qlist>ol>li:last-child>:last-child,.sidebarblock>.content>:last-child>:last-child,.sidebarblock>.content .olist>ol>li:last-child>:last-child,.sidebarblock>.content .ulist>ul>li:last-child>:last-child,.sidebarblock>.content .qlist>ol>li:last-child>:last-child{margin-bottom:0}
.literalblock pre,.listingblock>.content>pre{-webkit-border-radius:4px;border-radius:4px;overflow-x:auto;padding:1em;font-size:.8125em}
@media screen and (min-width:768px){.literalblock pre,.listingblock>.content>pre{font-size:.90625em}}
@media screen and (min-width:1280px){.literalblock pre,.listingblock>.content>pre{font-size:1em}}
.literalblock pre,.listingblock>.content>pre:not(.highlight),.listingblock>.content>pre[class="highlight"],.listingblock>.content>pre[class^="highlight "]{background:#f7f7f8}
.literalblock.output pre{color:#f7f7f8;background:rgba(0,0,0,.9)}
.listingblock>.content{position:relative}
.listingblock code[data-lang]::before{display:none;content:attr(data-lang);position:absolute;font-size:.75em;top:.425rem;right:.5rem;line-height:1;text-transform:uppercase;color:inherit;opacity:.5}
.listingblock:hover code[data-lang]::before{display:block}
.listingblock.terminal pre .command::before{content:attr(data-prompt);padding-right:.5em;color:inherit;opacity:.5}
.listingblock.terminal pre .command:not([data-prompt])::before{content:"$"}
.listingblock pre.highlightjs{padding:0}
.listingblock pre.highlightjs>code{padding:1em;-webkit-border-radius:4px;border-radius:4px}
.listingblock pre.prettyprint{border-width:0}
.prettyprint{background:#f7f7f8}
pre.prettyprint .linenums{line-height:1.45;margin-left:2em}
pre.prettyprint li{background:none;list-style-type:inherit;padding-left:0}
pre.prettyprint li code[data-lang]::before{opacity:1}
pre.prettyprint li:not(:first-child) code[data-lang]::before{display:none}
table.linenotable{border-collapse:separate;border:0;margin-bottom:0;background:none}
table.linenotable td[class]{color:inherit;vertical-align:top;padding:0;line-height:inherit;white-space:normal}
table.linenotable td.code{padding-left:.75em}
table.linenotable td.linenos{border-right:1px solid currentColor;opacity:.35;padding-right:.5em}
pre.pygments .lineno{border-right:1px solid currentColor;opacity:.35;display:inline-block;margin-right:.75em}
pre.pygments .lineno::before{content:"";margin-right:-.125em}
.quoteblock{margin:0 1em 1.25em 1.5em;display:table}
.quoteblock:not(.excerpt)>.title{margin-left:-1.5em;margin-bottom:.75em}
.quoteblock blockquote,.quoteblock p{color:rgba(0,0,0,.85);font-size:1.15rem;line-height:1.75;word-spacing:.1em;letter-spacing:0;font-style:italic;text-align:justify}
.quoteblock blockquote{margin:0;padding:0;border:0}
.quoteblock blockquote::before{content:"\201c";float:left;font-size:2.75em;font-weight:bold;line-height:.6em;margin-left:-.6em;color:#7a2518;text-shadow:0 1px 2px rgba(0,0,0,.1)}
.quoteblock blockquote>.paragraph:last-child p{margin-bottom:0}
.quoteblock .attribution{margin-top:.75em;margin-right:.5ex;text-align:right}
.verseblock{margin:0 1em 1.25em}
.verseblock pre{font-family:"Open Sans","DejaVu Sans",sans;font-size:1.15rem;color:rgba(0,0,0,.85);font-weight:300;text-rendering:optimizeLegibility}
.verseblock pre strong{font-weight:400}
.verseblock .attribution{margin-top:1.25rem;margin-left:.5ex}
.quoteblock .attribution,.verseblock .attribution{font-size:.9375em;line-height:1.45;font-style:italic}
.quoteblock .attribution br,.verseblock .attribution br{display:none}
.quoteblock .attribution cite,.verseblock .attribution cite{display:block;letter-spacing:-.025em;color:rgba(0,0,0,.6)}
.quoteblock.abstract blockquote::before,.quoteblock.excerpt blockquote::before,.quoteblock .quoteblock blockquote::before{display:none}
.quoteblock.abstract blockquote,.quoteblock.abstract p,.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{line-height:1.6;word-spacing:0}
.quoteblock.abstract{margin:0 1em 1.25em;display:block}
.quoteblock.abstract>.title{margin:0 0 .375em;font-size:1.15em;text-align:center}
.quoteblock.excerpt>blockquote,.quoteblock .quoteblock{padding:0 0 .25em 1em;border-left:.25em solid #dddddf}
.quoteblock.excerpt,.quoteblock .quoteblock{margin-left:0}
.quoteblock.excerpt blockquote,.quoteblock.excerpt p,.quoteblock .quoteblock blockquote,.quoteblock .quoteblock p{color:inherit;font-size:1.0625rem}
.quoteblock.excerpt .attribution,.quoteblock .quoteblock .attribution{color:inherit;text-align:left;margin-right:0}
table.tableblock{max-width:100%}
p.tableblock:last-child{margin-bottom:0}
td.tableblock>.content{margin-bottom:1.25em}
td.tableblock>.content>:last-child{margin-bottom:-1.25em}
table.tableblock,th.tableblock,td.tableblock{border:0 solid #dedede}
table.grid-all>*>tr>*{border-width:1px}
table.grid-cols>*>tr>*{border-width:0 1px}
table.grid-rows>*>tr>*{border-width:1px 0}
table.frame-all{border-width:1px}
table.frame-ends{border-width:1px 0}
table.frame-sides{border-width:0 1px}
table.frame-none>colgroup+*>:first-child>*,table.frame-sides>colgroup+*>:first-child>*{border-top-width:0}
table.frame-none>:last-child>:last-child>*,table.frame-sides>:last-child>:last-child>*{border-bottom-width:0}
table.frame-none>*>tr>:first-child,table.frame-ends>*>tr>:first-child{border-left-width:0}
table.frame-none>*>tr>:last-child,table.frame-ends>*>tr>:last-child{border-right-width:0}
table.stripes-all tr,table.stripes-odd tr:nth-of-type(odd),table.stripes-even tr:nth-of-type(even),table.stripes-hover tr:hover{background:#f8f8f7}
th.halign-left,td.halign-left{text-align:left}
th.halign-right,td.halign-right{text-align:right}
th.halign-center,td.halign-center{text-align:center}
th.valign-top,td.valign-top{vertical-align:top}
th.valign-bottom,td.valign-bottom{vertical-align:bottom}
th.valign-middle,td.valign-middle{vertical-align:middle}
table thead th,table tfoot th{font-weight:bold}
tbody tr th{display:table-cell;line-height:1.6;background:#f7f8f7}
tbody tr th,tbody tr th p,tfoot tr th,tfoot tr th p{color:rgba(0,0,0,.8);font-weight:bold}
p.tableblock>code:only-child{background:none;padding:0}
p.tableblock{font-size:1em}
ol{margin-left:1.75em}
ul li ol{margin-left:1.5em}
dl dd{margin-left:1.125em}
dl dd:last-child,dl dd:last-child>:last-child{margin-bottom:0}
ol>li p,ul>li p,ul dd,ol dd,.olist .olist,.ulist .ulist,.ulist .olist,.olist .ulist{margin-bottom:.625em}
ul.checklist,ul.none,ol.none,ul.no-bullet,ol.no-bullet,ol.unnumbered,ul.unstyled,ol.unstyled{list-style-type:none}
ul.no-bullet,ol.no-bullet,ol.unnumbered{margin-left:.625em}
ul.unstyled,ol.unstyled{margin-left:0}
ul.checklist{margin-left:.625em}
ul.checklist li>p:first-child>.fa-square-o:first-child,ul.checklist li>p:first-child>.fa-check-square-o:first-child{width:1.25em;font-size:.8em;position:relative;bottom:.125em}
ul.checklist li>p:first-child>input[type="checkbox"]:first-child{margin-right:.25em}
ul.inline{display:-ms-flexbox;display:-webkit-box;display:flex;-ms-flex-flow:row wrap;-webkit-flex-flow:row wrap;flex-flow:row wrap;list-style:none;margin:0 0 .625em -1.25em}
ul.inline>li{margin-left:1.25em}
.unstyled dl dt{font-weight:400;font-style:normal}
ol.arabic{list-style-type:decimal}
ol.decimal{list-style-type:decimal-leading-zero}
ol.loweralpha{list-style-type:lower-alpha}
ol.upperalpha{list-style-type:upper-alpha}
ol.lowerroman{list-style-type:lower-roman}
ol.upperroman{list-style-type:upper-roman}
ol.lowergreek{list-style-type:lower-greek}
.hdlist>table,.colist>table{border:0;background:none}
.hdlist>table>tbody>tr,.colist>table>tbody>tr{background:none}
td.hdlist1,td.hdlist2{vertical-align:top;padding:0 .625em}
td.hdlist1{font-weight:bold;padding-bottom:1.25em}
.literalblock+.colist,.listingblock+.colist{margin-top:-.5em}
.colist td:not([class]):first-child{padding:.4em .75em 0;line-height:1;vertical-align:top}
.colist td:not([class]):first-child img{max-width:none}
.colist td:not([class]):last-child{padding:.25em 0}
.thumb,.th{line-height:0;display:inline-block;border:solid 4px #fff;-webkit-box-shadow:0 0 0 1px #ddd;box-shadow:0 0 0 1px #ddd}
.imageblock.left{margin:.25em .625em 1.25em 0}
.imageblock.right{margin:.25em 0 1.25em .625em}
.imageblock>.title{margin-bottom:0}
.imageblock.thumb,.imageblock.th{border-width:6px}
.imageblock.thumb>.title,.imageblock.th>.title{padding:0 .125em}
.image.left,.image.right{margin-top:.25em;margin-bottom:.25em;display:inline-block;line-height:0}
.image.left{margin-right:.625em}
.image.right{margin-left:.625em}
a.image{text-decoration:none;display:inline-block}
a.image object{pointer-events:none}
sup.footnote,sup.footnoteref{font-size:.875em;position:static;vertical-align:super}
sup.footnote a,sup.footnoteref a{text-decoration:none}
sup.footnote a:active,sup.footnoteref a:active{text-decoration:underline}
#footnotes{padding-top:.75em;padding-bottom:.75em;margin-bottom:.625em}
#footnotes hr{width:20%;min-width:6.25em;margin:-.25em 0 .75em;border-width:1px 0 0}
#footnotes .footnote{padding:0 .375em 0 .225em;line-height:1.3334;font-size:.875em;margin-left:1.2em;margin-bottom:.2em}
#footnotes .footnote a:first-of-type{font-weight:bold;text-decoration:none;margin-left:-1.05em}
#footnotes .footnote:last-of-type{margin-bottom:0}
#content #footnotes{margin-top:-.625em;margin-bottom:0;padding:.75em 0}
.gist .file-data>table{border:0;background:#fff;width:100%;margin-bottom:0}
.gist .file-data>table td.line-data{width:99%}
div.unbreakable{page-break-inside:avoid}
.big{font-size:larger}
.small{font-size:smaller}
.underline{text-decoration:underline}
.overline{text-decoration:overline}
.line-through{text-decoration:line-through}
.aqua{color:#00bfbf}
.aqua-background{background:#00fafa}
.black{color:#000}
.black-background{background:#000}
.blue{color:#0000bf}
.blue-background{background:#0000fa}
.fuchsia{color:#bf00bf}
.fuchsia-background{background:#fa00fa}
.gray{color:#606060}
.gray-background{background:#7d7d7d}
.green{color:#006000}
.green-background{background:#007d00}
.lime{color:#00bf00}
.lime-background{background:#00fa00}
.maroon{color:#600000}
.maroon-background{background:#7d0000}
.navy{color:#000060}
.navy-background{background:#00007d}
.olive{color:#606000}
.olive-background{background:#7d7d00}
.purple{color:#600060}
.purple-background{background:#7d007d}
.red{color:#bf0000}
.red-background{background:#fa0000}
.silver{color:#909090}
.silver-background{background:#bcbcbc}
.teal{color:#006060}
.teal-background{background:#007d7d}
.white{color:#bfbfbf}
.white-background{background:#fafafa}
.yellow{color:#bfbf00}
.yellow-background{background:#fafa00}
span.icon>.fa{cursor:default}
a span.icon>.fa{cursor:inherit}
.admonitionblock td.icon [class^="fa icon-"]{font-size:2.5em;text-shadow:1px 1px 2px rgba(0,0,0,.5);cursor:default}
.admonitionblock td.icon .icon-note::before{content:"\f05a";color:#19407c}
.admonitionblock td.icon .icon-tip::before{content:"\f0eb";text-shadow:1px 1px 2px rgba(155,155,0,.8);color:#111}
.admonitionblock td.icon .icon-warning::before{content:"\f071";color:#bf6900}
.admonitionblock td.icon .icon-caution::before{content:"\f06d";color:#bf3400}
.admonitionblock td.icon .icon-important::before{content:"\f06a";color:#bf0000}
.conum[data-value]{display:inline-block;color:#fff!important;background:rgba(0,0,0,.8);-webkit-border-radius:50%;border-radius:50%;text-align:center;font-size:.75em;width:1.67em;height:1.67em;line-height:1.67em;font-family:"Open Sans","DejaVu Sans",sans-serif;font-style:normal;font-weight:bold}
.conum[data-value] *{color:#fff!important}
.conum[data-value]+b{display:none}
.conum[data-value]::after{content:attr(data-value)}
pre .conum[data-value]{position:relative;top:-.125em}
b.conum *{color:inherit!important}
.conum:not([data-value]):empty{display:none}
dt,th.tableblock,td.content,div.footnote{text-rendering:optimizeLegibility}
h1,h2,p,td.content,span.alt{letter-spacing:-.01em}
p strong,td.content strong,div.footnote strong{letter-spacing:-.005em}
p,blockquote,dt,td.content,span.alt{font-size:1.0625rem}
p{margin-bottom:1.25rem}
.sidebarblock p,.sidebarblock dt,.sidebarblock td.content,p.tableblock{font-size:1em}
.exampleblock>.content{background:#fffef7;border-color:#e0e0dc;-webkit-box-shadow:0 1px 4px #e0e0dc;box-shadow:0 1px 4px #e0e0dc}
.print-only{display:none!important}
@page{margin:1.25cm .75cm}
@media print{*{-webkit-box-shadow:none!important;box-shadow:none!important;text-shadow:none!important}
html{font-size:80%}
a{color:inherit!important;text-decoration:underline!important}
a.bare,a[href^="#"],a[href^="mailto:"]{text-decoration:none!important}
a[href^="http:"]:not(.bare)::after,a[href^="https:"]:not(.bare)::after{content:"(" attr(href) ")";display:inline-block;font-size:.875em;padding-left:.25em}
abbr[title]::after{content:" (" attr(title) ")"}
pre,blockquote,tr,img,object,svg{page-break-inside:avoid}
thead{display:table-header-group}
svg{max-width:100%}
p,blockquote,dt,td.content{font-size:1em;orphans:3;widows:3}
h2,h3,#toctitle,.sidebarblock>.content>.title{page-break-after:avoid}
#header,#content,#footnotes,#footer{max-width:none}
#toc,.sidebarblock,.exampleblock>.content{background:none!important}
#toc{border-bottom:1px solid #dddddf!important;padding-bottom:0!important}
body.book #header{text-align:center}
body.book #header>h1:first-child{border:0!important;margin:2.5em 0 1em}
body.book #header .details{border:0!important;display:block;padding:0!important}
body.book #header .details span:first-child{margin-left:0!important}
body.book #header .details br{display:block}
body.book #header .details br+span::before{content:none!important}
body.book #toc{border:0!important;text-align:left!important;padding:0!important;margin:0!important}
body.book #toc,body.book #preamble,body.book h1.sect0,body.book .sect1>h2{page-break-before:always}
.listingblock code[data-lang]::before{display:block}
#footer{padding:0 .9375em}
.hide-on-print{display:none!important}
.print-only{display:block!important}
.hide-for-print{display:none!important}
.show-for-print{display:inherit!important}}
@media print,amzn-kf8{#header>h1:first-child{margin-top:1.25rem}
.sect1{padding:0!important}
.sect1+.sect1{border:0}
#footer{background:none}
#footer-text{color:rgba(0,0,0,.6);font-size:.9em}}
@media amzn-kf8{#header,#content,#footnotes,#footer{padding:0}}
</style>
<style>
pre{ white-space:pre }
#header,#content,#footnotes,#footer {
  margin-left:40px;
  padding-right:60px;
  max-width:none;
}
</style>
</head>
<body class="article">
<div id="header">
<h1>Linux Kernel Module Cheat</h1>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://zenodo.org/badge/latestdoi/64534859"><span class="image"><img src="https://zenodo.org/badge/64534859.svg" alt="64534859"></span></a></p>
</div>
<div class="paragraph">
<p>The perfect emulation setup to study and develop the <a href="#linux-kernel">Linux kernel</a> v5.9.2, kernel modules, <a href="#qemu-buildroot-setup">QEMU</a>, <a href="#gem5-buildroot-setup">gem5</a> and x86_64, ARMv7 and ARMv8 <a href="#userland-assembly">userland</a> and <a href="#baremetal-setup">baremetal</a> assembly, <a href="#c">ANSI C</a>, <a href="#cpp">C++</a> and <a href="#posix">POSIX</a>. <a href="#gdb">GDB step debug</a> and <a href="#kgdb">KGDB</a> just work. Powered by <a href="#about-the-qemu-buildroot-setup">Buildroot</a> and <a href="#about-the-baremetal-setup">crosstool-NG</a>. Highly automated. Thoroughly documented. Automated <a href="#test-this-repo">tests</a>. "Tested" in an Ubuntu 20.04 host.</p>
</div>
<div class="paragraph">
<p><a href="https://twitter.com/dakami/status/1344853681749934080">Dan Kaminski-approved</a>™ <a href="https://en.wikipedia.org/wiki/Dan_Kaminsky">RIP</a>.</p>
</div>
<div class="paragraph">
<p>TL;DR: <a href="#qemu-buildroot-setup-getting-started">Section 2.2.1, &#8220;QEMU Buildroot setup getting started&#8221;</a></p>
</div>
<div class="paragraph">
<p>The source code for this page is located at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat</a>. Due to <a href="https://github.com/isaacs/github/issues/1610">a GitHub limitation</a>, this README is too long and not fully rendered on github.com, so either use:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://cirosantilli.com/linux-kernel-module-cheat" class="bare">https://cirosantilli.com/linux-kernel-module-cheat</a></p>
</li>
<li>
<p><a href="https://cirosantilli.com/linux-kernel-module-cheat/index-split" class="bare">https://cirosantilli.com/linux-kernel-module-cheat/index-split</a>: split header version</p>
</li>
<li>
<p><a href="#build-the-documentation">build the docs yourself</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/china-dictatorship" class="bare">https://github.com/cirosantilli/china-dictatorship</a> | <a href="https://cirosantilli.com/china-dictatorship/xinjiang" class="bare">https://cirosantilli.com/china-dictatorship/xinjiang</a></p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/china-dictatorship-media/master/Xinjiang_prisoners_sitting_identified.jpeg" alt="Xinjiang prisoners sitting identified" width="800">
</div>
</div>
<div id="toc" class="toc">
<div id="toctitle" class="title"></div>
<ul class="sectlevel1">
<li><a href="#china">1. <code>--china</code></a></li>
<li><a href="#getting-started">2. Getting started</a>
<ul class="sectlevel2">
<li><a href="#should-you-waste-your-life-with-systems-programming">2.1. Should you waste your life with systems programming?</a></li>
<li><a href="#qemu-buildroot-setup">2.2. QEMU Buildroot setup</a>
<ul class="sectlevel3">
<li><a href="#qemu-buildroot-setup-getting-started">2.2.1. QEMU Buildroot setup getting started</a></li>
<li><a href="#how-to-hack-stuff">2.2.2. How to hack stuff</a>
<ul class="sectlevel4">
<li><a href="#your-first-linux-kernel-hack">2.2.2.1. Your first Linux kernel hack</a></li>
<li><a href="#your-first-kernel-module-hack">2.2.2.2. Your first kernel module hack</a></li>
<li><a href="#your-first-glibc-hack">2.2.2.3. Your first glibc hack</a></li>
<li><a href="#your-first-binutils-hack">2.2.2.4. Your first Binutils hack</a></li>
<li><a href="#your-first-gcc-hack">2.2.2.5. Your first GCC hack</a></li>
</ul>
</li>
<li><a href="#about-the-qemu-buildroot-setup">2.2.3. About the QEMU Buildroot setup</a></li>
</ul>
</li>
<li><a href="#dry-run">2.3. Dry run to get commands for your project</a></li>
<li><a href="#gem5-buildroot-setup">2.4. gem5 Buildroot setup</a>
<ul class="sectlevel3">
<li><a href="#about-the-gem5-buildroot-setup">2.4.1. About the gem5 Buildroot setup</a></li>
<li><a href="#gem5-buildroot-setup-getting-started">2.4.2. gem5 Buildroot setup getting started</a></li>
</ul>
</li>
<li><a href="#docker">2.5. Docker host setup</a></li>
<li><a href="#prebuilt">2.6. Prebuilt setup</a>
<ul class="sectlevel3">
<li><a href="#about-the-prebuilt-setup">2.6.1. About the prebuilt setup</a></li>
<li><a href="#prebuilt-setup-getting-started">2.6.2. Prebuilt setup getting started</a></li>
</ul>
</li>
<li><a href="#host">2.7. Host kernel module setup</a>
<ul class="sectlevel3">
<li><a href="#hello-host">2.7.1. Hello host</a></li>
</ul>
</li>
<li><a href="#userland-setup">2.8. Userland setup</a>
<ul class="sectlevel3">
<li><a href="#about-the-userland-setup">2.8.1. About the userland setup</a></li>
<li><a href="#userland-setup-getting-started">2.8.2. Userland setup getting started</a>
<ul class="sectlevel4">
<li><a href="#userland-setup-getting-started-natively">2.8.2.1. Userland setup getting started natively</a></li>
<li><a href="#userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode">2.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode</a></li>
<li><a href="#userland-setup-getting-started-full-system">2.8.2.3. Userland setup getting started full system</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#baremetal-setup">2.9. Baremetal setup</a>
<ul class="sectlevel3">
<li><a href="#about-the-baremetal-setup">2.9.1. About the baremetal setup</a></li>
<li><a href="#baremetal-setup-getting-started">2.9.2. Baremetal setup getting started</a></li>
</ul>
</li>
<li><a href="#build-the-documentation">2.10. Build the documentation</a></li>
</ul>
</li>
<li><a href="#gdb">3. GDB step debug</a>
<ul class="sectlevel2">
<li><a href="#gdb-step-debug-kernel-boot">3.1. GDB step debug kernel boot</a>
<ul class="sectlevel3">
<li><a href="#gdb-step-debug-kernel-boot-other-archs">3.1.1. GDB step debug kernel boot other archs</a></li>
<li><a href="#kernel-o0">3.1.2. Disable kernel compiler optimizations</a></li>
</ul>
</li>
<li><a href="#gdb-step-debug-kernel-post-boot">3.2. GDB step debug kernel post-boot</a></li>
<li><a href="#tmux">3.3. tmux</a>
<ul class="sectlevel3">
<li><a href="#tmux-gem5">3.3.1. tmux gem5</a></li>
</ul>
</li>
<li><a href="#gdb-step-debug-kernel-module">3.4. GDB step debug kernel module</a>
<ul class="sectlevel3">
<li><a href="#gdb-step-debug-kernel-module-arm">3.4.1. GDB step debug kernel module insmodded by init on ARM</a></li>
<li><a href="#gdb-module-init">3.4.2. GDB module_init</a>
<ul class="sectlevel4">
<li><a href="#gdb-module-init-step-into-it">3.4.2.1. GDB module_init step into it</a></li>
<li><a href="#gdb-module-init-calculate-entry-address">3.4.2.2. GDB module_init calculate entry address</a></li>
<li><a href="#gdb-module-init-break-at-the-end-of-sys-init-module">3.4.2.3. GDB module_init break at the end of sys_init_module</a></li>
<li><a href="#gdb-module-init-add-trap-instruction">3.4.2.4. GDB module_init add trap instruction</a></li>
</ul>
</li>
<li><a href="#bypass-lx-symbols">3.4.3. Bypass lx-symbols</a></li>
</ul>
</li>
<li><a href="#gdb-step-debug-early-boot">3.5. GDB step debug early boot</a>
<ul class="sectlevel3">
<li><a href="#linux-kernel-entry-point">3.5.1. Linux kernel entry point</a>
<ul class="sectlevel4">
<li><a href="#arm64-secondary-cpu-entry-point">3.5.1.1. arm64 secondary CPU entry point</a></li>
</ul>
</li>
<li><a href="#linux-kernel-arch-agnostic-entry-point">3.5.2. Linux kernel arch-agnostic entry point</a></li>
<li><a href="#linux-kernel-early-boot-messages">3.5.3. Linux kernel early boot messages</a></li>
</ul>
</li>
<li><a href="#gdb-step-debug-userland-processes">3.6. GDB step debug userland processes</a>
<ul class="sectlevel3">
<li><a href="#gdb-step-debug-userland-custom-init">3.6.1. GDB step debug userland custom init</a></li>
<li><a href="#gdb-step-debug-userland-busybox-init">3.6.2. GDB step debug userland BusyBox init</a></li>
<li><a href="#gdb-step-debug-userland-non-init">3.6.3. GDB step debug userland non-init</a>
<ul class="sectlevel4">
<li><a href="#gdb-step-debug-userland-non-init-without-gdb-wait">3.6.3.1. GDB step debug userland non-init without --gdb-wait</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gdb-call">3.7. GDB call</a></li>
<li><a href="#gdb-view-arm-system-registers">3.8. GDB view ARM system registers</a></li>
<li><a href="#gdb-step-debug-multicore-userland">3.9. GDB step debug multicore userland</a></li>
<li><a href="#linux-kernel-gdb-scripts">3.10. Linux kernel GDB scripts</a>
<ul class="sectlevel3">
<li><a href="#lx-ps">3.10.1. lx-ps</a>
<ul class="sectlevel4">
<li><a href="#config-pid-in-contextidr">3.10.1.1. CONFIG_PID_IN_CONTEXTIDR</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#debug-the-gdb-remote-protocol">3.11. Debug the GDB remote protocol</a>
<ul class="sectlevel3">
<li><a href="#remote-g-packet">3.11.1. Remote 'g' packet reply is too long</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#kgdb">4. KGDB</a>
<ul class="sectlevel2">
<li><a href="#kgdb-arm">4.1. KGDB ARM</a></li>
<li><a href="#kgdb-kernel-modules">4.2. KGDB kernel modules</a></li>
<li><a href="#kdb">4.3. KDB</a>
<ul class="sectlevel3">
<li><a href="#kdb-graphic">4.3.1. KDB graphic</a></li>
<li><a href="#kdb-arm">4.3.2. KDB ARM</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gdbserver">5. gdbserver</a>
<ul class="sectlevel2">
<li><a href="#gdbserver-busybox">5.1. gdbserver BusyBox</a></li>
<li><a href="#gdbserver-libc">5.2. gdbserver libc</a></li>
<li><a href="#gdbserver-dynamic-loader">5.3. gdbserver dynamic loader</a></li>
</ul>
</li>
<li><a href="#cpu-architecture">6. CPU architecture</a>
<ul class="sectlevel2">
<li><a href="#x86-64">6.1. x86_64</a>
<ul class="sectlevel3">
<li><a href="#ring0">6.1.1. ring0</a></li>
</ul>
</li>
<li><a href="#arm">6.2. arm</a>
<ul class="sectlevel3">
<li><a href="#run-arm-executable-in-aarch64">6.2.1. Run arm executable in aarch64</a></li>
</ul>
</li>
<li><a href="#mips">6.3. MIPS</a></li>
<li><a href="#other-architectures">6.4. Other architectures</a></li>
</ul>
</li>
<li><a href="#init">7. init</a>
<ul class="sectlevel2">
<li><a href="#replace-init">7.1. Replace init</a>
<ul class="sectlevel3">
<li><a href="#poweroff-out">7.1.1. poweroff.out</a></li>
<li><a href="#sleep-forever-out">7.1.2. sleep_forever.out</a></li>
<li><a href="#time-boot-out">7.1.3. time_boot.out</a></li>
</ul>
</li>
<li><a href="#init-busybox">7.2. Run command at the end of BusyBox init</a></li>
<li><a href="#path-to-init">7.3. Path to init</a></li>
<li><a href="#init-environment">7.4. Init environment</a>
<ul class="sectlevel3">
<li><a href="#init-arguments">7.4.1. init arguments</a></li>
<li><a href="#init-environment-env">7.4.2. init environment env</a></li>
<li><a href="#busybox-shell-init-environment">7.4.3. BusyBox shell init environment</a>
<ul class="sectlevel4">
<li><a href="#busybox-shell-initrc-files">7.4.3.1. BusyBox shell initrc files</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#initrd">8. initrd</a>
<ul class="sectlevel2">
<li><a href="#initrd-in-desktop-distros">8.1. initrd in desktop distros</a></li>
<li><a href="#initramfs">8.2. initramfs</a></li>
<li><a href="#rootfs">8.3. rootfs</a>
<ul class="sectlevel3">
<li><a href="#devroot">8.3.1. /dev/root</a></li>
</ul>
</li>
<li><a href="#gem5-initrd">8.4. gem5 initrd</a></li>
<li><a href="#gem5-initramfs">8.5. gem5 initramfs</a></li>
</ul>
</li>
<li><a href="#device-tree">9. Device tree</a>
<ul class="sectlevel2">
<li><a href="#dtb-files">9.1. DTB files</a></li>
<li><a href="#device-tree-syntax">9.2. Device tree syntax</a></li>
<li><a href="#get-device-tree-from-a-running-kernel">9.3. Get device tree from a running kernel</a></li>
<li><a href="#device-tree-emulator-generation">9.4. Device tree emulator generation</a></li>
</ul>
</li>
<li><a href="#kvm">10. KVM</a>
<ul class="sectlevel2">
<li><a href="#kvm-arm">10.1. KVM arm</a></li>
<li><a href="#gem5-kvm">10.2. gem5 KVM</a></li>
</ul>
</li>
<li><a href="#user-mode-simulation">11. User mode simulation</a>
<ul class="sectlevel2">
<li><a href="#qemu-user-mode-getting-started">11.1. QEMU user mode getting started</a>
<ul class="sectlevel3">
<li><a href="#user-mode-gdb">11.1.1. User mode GDB</a></li>
</ul>
</li>
<li><a href="#user-mode-tests">11.2. User mode tests</a></li>
<li><a href="#user-mode-buildroot-executables">11.3. User mode Buildroot executables</a></li>
<li><a href="#user-mode-simulation-with-glibc">11.4. User mode simulation with glibc</a>
<ul class="sectlevel3">
<li><a href="#fatal-kernel-too-old-failure-in-userland-simulation">11.4.1. FATAL: kernel too old failure in userland simulation</a></li>
<li><a href="#stack-smashing-detected-when-using-glibc">11.4.2. stack smashing detected when using glibc</a></li>
</ul>
</li>
<li><a href="#user-mode-static-executables">11.5. User mode static executables</a>
<ul class="sectlevel3">
<li><a href="#user-mode-static-executables-with-dynamic-libraries">11.5.1. User mode static executables with dynamic libraries</a>
<ul class="sectlevel4">
<li><a href="#cpp-static-and-pthreads">11.5.1.1. C++ static and pthreads</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#syscall-emulation-mode-program-stdin">11.6. syscall emulation mode program stdin</a></li>
<li><a href="#gem5-syscall-emulation-mode">11.7. gem5 syscall emulation mode</a>
<ul class="sectlevel3">
<li><a href="#gem5-dynamic-linked-executables-in-syscall-emulation">11.7.1. gem5 dynamic linked executables in syscall emulation</a></li>
<li><a href="#gem5-syscall-emulation-exit-status">11.7.2. gem5 syscall emulation exit status</a></li>
<li><a href="#gem5-syscall-emulation-mode-syscall-tracing">11.7.3. gem5 syscall emulation mode syscall tracing</a></li>
<li><a href="#gem5-syscall-emulation-multithreading">11.7.4. gem5 syscall emulation multithreading</a></li>
<li><a href="#gem5-syscall-emulation-multiple-executables">11.7.5. gem5 syscall emulation multiple executables</a>
<ul class="sectlevel4">
<li><a href="#gem5-syscall-emulation-smt">11.7.5.1. gem5 syscall emulation --smt</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#qemu-user-mode-quirks">11.8. QEMU user mode quirks</a>
<ul class="sectlevel3">
<li><a href="#qemu-user-mode-does-not-show-stdout-immediately">11.8.1. QEMU user mode does not show stdout immediately</a>
<ul class="sectlevel4">
<li><a href="#qemu-user-mode-does-not-show-errors">11.8.1.1. QEMU user mode does not show errors</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#kernel-module-utilities">12. Kernel module utilities</a>
<ul class="sectlevel2">
<li><a href="#insmod">12.1. insmod</a></li>
<li><a href="#myinsmod">12.2. myinsmod</a></li>
<li><a href="#modprobe">12.3. modprobe</a></li>
<li><a href="#kmod">12.4. kmod</a>
<ul class="sectlevel3">
<li><a href="#module-init-tools">12.4.1. module-init-tools</a></li>
<li><a href="#kmod-modprobe">12.4.2. kmod modprobe</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#filesystems">13. Filesystems</a>
<ul class="sectlevel2">
<li><a href="#overlayfs">13.1. OverlayFS</a></li>
<li><a href="#secondary-disk">13.2. Secondary disk</a></li>
</ul>
</li>
<li><a href="#graphics">14. Graphics</a>
<ul class="sectlevel2">
<li><a href="#qemu-text-mode">14.1. QEMU text mode</a>
<ul class="sectlevel3">
<li><a href="#quit-qemu-from-text-mode">14.1.1. Quit QEMU from text mode</a></li>
</ul>
</li>
<li><a href="#qemu-graphic-mode">14.2. QEMU graphic mode</a>
<ul class="sectlevel3">
<li><a href="#scroll-up-in-graphic-mode">14.2.1. Scroll up in graphic mode</a></li>
<li><a href="#qemu-graphic-mode-arm">14.2.2. QEMU Graphic mode arm</a>
<ul class="sectlevel4">
<li><a href="#qemu-graphic-mode-arm-terminal">14.2.2.1. QEMU graphic mode arm terminal</a></li>
<li><a href="#qemu-graphic-mode-arm-terminal-implementation">14.2.2.2. QEMU graphic mode arm terminal implementation</a></li>
<li><a href="#qemu-graphic-mode-arm-vga">14.2.2.3. QEMU graphic mode arm VGA</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-graphic-mode">14.3. gem5 graphic mode</a>
<ul class="sectlevel3">
<li><a href="#graphic-mode-gem5-aarch64">14.3.1. Graphic mode gem5 aarch64</a></li>
<li><a href="#gem5-graphic-mode-dp650">14.3.2. gem5 graphic mode DP650</a></li>
<li><a href="#gem5-graphic-mode-internals">14.3.3. gem5 graphic mode internals</a></li>
</ul>
</li>
<li><a href="#x11">14.4. X11 Buildroot</a>
<ul class="sectlevel3">
<li><a href="#x11-buildroot-mouse-not-moving">14.4.1. X11 Buildroot mouse not moving</a></li>
<li><a href="#x11-buildroot-arm">14.4.2. X11 Buildroot ARM</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#networking">15. Networking</a>
<ul class="sectlevel2">
<li><a href="#enable-networking">15.1. Enable networking</a></li>
<li><a href="#ping">15.2. ping</a></li>
<li><a href="#guest-host-networking">15.3. Guest host networking</a>
<ul class="sectlevel3">
<li><a href="#host-to-guest-networking">15.3.1. Host to guest networking</a>
<ul class="sectlevel4">
<li><a href="#nc-host-to-guest">15.3.1.1. nc host to guest</a></li>
<li><a href="#ssh-into-guest">15.3.1.2. ssh into guest</a></li>
<li><a href="#gem5-host-to-guest-networking">15.3.1.3. gem5 host to guest networking</a></li>
</ul>
</li>
<li><a href="#guest-to-host-networking">15.3.2. Guest to host networking</a></li>
</ul>
</li>
<li><a href="#9p">15.4. 9P</a>
<ul class="sectlevel3">
<li><a href="#9p-vs-nfs">15.4.1. 9P vs NFS</a></li>
<li><a href="#9p-getting-started">15.4.2. 9P getting started</a></li>
<li><a href="#gem5-9p">15.4.3. gem5 9P</a></li>
<li><a href="#nfs">15.4.4. NFS</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#operating-systems">16. Operating systems</a></li>
<li><a href="#linux-kernel">17. Linux kernel</a>
<ul class="sectlevel2">
<li><a href="#linux-kernel-configuration">17.1. Linux kernel configuration</a>
<ul class="sectlevel3">
<li><a href="#modify-kernel-config">17.1.1. Modify kernel config</a></li>
<li><a href="#find-the-kernel-config">17.1.2. Find the kernel config</a></li>
<li><a href="#kernel-configs-about">17.1.3. About our Linux kernel configs</a>
<ul class="sectlevel4">
<li><a href="#buildroot-kernel-config">17.1.3.1. About Buildroot&#8217;s kernel configs</a>
<ul class="sectlevel5">
<li><a href="#linux-kernel-defconfig">17.1.3.1.1. Linux kernel defconfig</a></li>
<li><a href="#linux-kernel-min-config">17.1.3.1.2. Linux kernel min config</a></li>
</ul>
</li>
<li><a href="#notable-alternate-gem5-kernel-configs">17.1.3.2. Notable alternate gem5 kernel configs</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#kernel-version">17.2. Kernel version</a>
<ul class="sectlevel3">
<li><a href="#find-the-kernel-version">17.2.1. Find the kernel version</a></li>
<li><a href="#update-the-linux-kernel">17.2.2. Update the Linux kernel</a>
<ul class="sectlevel4">
<li><a href="#update-the-linux-kernel-lkmc-procedure">17.2.2.1. Update the Linux kernel LKMC procedure</a></li>
</ul>
</li>
<li><a href="#downgrade-the-linux-kernel">17.2.3. Downgrade the Linux kernel</a></li>
</ul>
</li>
<li><a href="#kernel-command-line-parameters">17.3. Kernel command line parameters</a>
<ul class="sectlevel3">
<li><a href="#kernel-command-line-parameters-escaping">17.3.1. Kernel command line parameters escaping</a></li>
<li><a href="#kernel-command-line-parameters-definition-points">17.3.2. Kernel command line parameters definition points</a></li>
<li><a href="#rw">17.3.3. rw</a></li>
<li><a href="#norandmaps">17.3.4. norandmaps</a></li>
</ul>
</li>
<li><a href="#printk">17.4. printk</a>
<ul class="sectlevel3">
<li><a href="#procsyskernelprintk">17.4.1. /proc/sys/kernel/printk</a></li>
<li><a href="#ignore-loglevel">17.4.2. ignore_loglevel</a></li>
<li><a href="#pr-debug">17.4.3. pr_debug</a>
<ul class="sectlevel4">
<li><a href="#pr-debug-is-different-from-printk-kern-debug">17.4.3.1. pr_debug != printk(KERN_DEBUG</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#kernel-module-apis">17.5. Kernel module APIs</a>
<ul class="sectlevel3">
<li><a href="#kernel-module-parameters">17.5.1. Kernel module parameters</a>
<ul class="sectlevel4">
<li><a href="#modprobe-conf">17.5.1.1. modprobe.conf</a></li>
</ul>
</li>
<li><a href="#kernel-module-dependencies">17.5.2. Kernel module dependencies</a>
<ul class="sectlevel4">
<li><a href="#kernel-module-dependencies-with-modprobe">17.5.2.1. Kernel module dependencies with modprobe</a></li>
</ul>
</li>
<li><a href="#module-info">17.5.3. MODULE_INFO</a></li>
<li><a href="#vermagic">17.5.4. vermagic</a></li>
<li><a href="#init-module">17.5.5. init_module</a></li>
<li><a href="#floating-point-in-kernel-modules">17.5.6. Floating point in kernel modules</a></li>
</ul>
</li>
<li><a href="#kernel-panic-and-oops">17.6. Kernel panic and oops</a>
<ul class="sectlevel3">
<li><a href="#kernel-panic">17.6.1. Kernel panic</a>
<ul class="sectlevel4">
<li><a href="#kernel-module-stack-trace-to-source-line">17.6.1.1. Kernel module stack trace to source line</a></li>
<li><a href="#bug-on">17.6.1.2. BUG_ON</a></li>
<li><a href="#exit-emulator-on-panic">17.6.1.3. Exit emulator on panic</a>
<ul class="sectlevel5">
<li><a href="#exit-qemu-on-panic">17.6.1.3.1. Exit QEMU on panic</a></li>
<li><a href="#exit-gem5-on-panic">17.6.1.3.2. Exit gem5 on panic</a></li>
</ul>
</li>
<li><a href="#reboot-on-panic">17.6.1.4. Reboot on panic</a></li>
<li><a href="#panic-trace-show-addresses-instead-of-symbols">17.6.1.5. Panic trace show addresses instead of symbols</a></li>
</ul>
</li>
<li><a href="#oops">17.6.2. Kernel oops</a></li>
<li><a href="#dump-stack">17.6.3. dump_stack</a></li>
<li><a href="#warn-on">17.6.4. WARN_ON</a></li>
<li><a href="#not-syncing-vfs">17.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)</a></li>
</ul>
</li>
<li><a href="#pseudo-filesystems">17.7. Pseudo filesystems</a>
<ul class="sectlevel3">
<li><a href="#debugfs">17.7.1. debugfs</a></li>
<li><a href="#procfs">17.7.2. procfs</a>
<ul class="sectlevel4">
<li><a href="#proc-version">17.7.2.1. /proc/version</a></li>
</ul>
</li>
<li><a href="#sysfs">17.7.3. sysfs</a></li>
<li><a href="#character-devices">17.7.4. Character devices</a>
<ul class="sectlevel4">
<li><a href="#automatically-create-character-device-file-on-insmod">17.7.4.1. Automatically create character device file on insmod</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#pseudo-files">17.8. Pseudo files</a>
<ul class="sectlevel3">
<li><a href="#file-operations">17.8.1. File operations</a></li>
<li><a href="#seq-file">17.8.2. seq_file</a>
<ul class="sectlevel4">
<li><a href="#seq-file-single-open">17.8.2.1. seq_file single_open</a></li>
</ul>
</li>
<li><a href="#poll">17.8.3. poll</a></li>
<li><a href="#ioctl">17.8.4. ioctl</a></li>
<li><a href="#mmap">17.8.5. mmap</a></li>
<li><a href="#anonymous-inode">17.8.6. Anonymous inode</a></li>
<li><a href="#netlink-sockets">17.8.7. netlink sockets</a></li>
</ul>
</li>
<li><a href="#kthread">17.9. kthread</a>
<ul class="sectlevel3">
<li><a href="#kthreads">17.9.1. kthreads</a></li>
<li><a href="#sleep">17.9.2. sleep</a></li>
<li><a href="#workqueues">17.9.3. Workqueues</a>
<ul class="sectlevel4">
<li><a href="#workqueue-from-workqueue">17.9.3.1. Workqueue from workqueue</a></li>
</ul>
</li>
<li><a href="#schedule">17.9.4. schedule</a></li>
<li><a href="#wait-queues">17.9.5. Wait queues</a></li>
</ul>
</li>
<li><a href="#timers">17.10. Timers</a></li>
<li><a href="#irq">17.11. IRQ</a>
<ul class="sectlevel3">
<li><a href="#irq-ko">17.11.1. irq.ko</a></li>
<li><a href="#dummy-irq">17.11.2. dummy-irq</a></li>
<li><a href="#procinterrupts">17.11.3. /proc/interrupts</a></li>
</ul>
</li>
<li><a href="#kernel-utility-functions">17.12. Kernel utility functions</a>
<ul class="sectlevel3">
<li><a href="#kstrto">17.12.1. kstrto</a></li>
<li><a href="#virt-to-phys">17.12.2. virt_to_phys</a>
<ul class="sectlevel4">
<li><a href="#userland-physical-address-experiments">17.12.2.1. Userland physical address experiments</a>
<ul class="sectlevel5">
<li><a href="#qemu-xp">17.12.2.1.1. QEMU xp</a></li>
<li><a href="#dev-mem">17.12.2.1.2. /dev/mem</a></li>
<li><a href="#pagemap-dump-out">17.12.2.1.3. pagemap_dump.out</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#linux-kernel-tracing">17.13. Linux kernel tracing</a>
<ul class="sectlevel3">
<li><a href="#config-proc-events">17.13.1. CONFIG_PROC_EVENTS</a>
<ul class="sectlevel4">
<li><a href="#config-proc-events-aarch64">17.13.1.1. CONFIG_PROC_EVENTS aarch64</a></li>
</ul>
</li>
<li><a href="#ftrace">17.13.2. ftrace</a>
<ul class="sectlevel4">
<li><a href="#ftrace-system-calls">17.13.2.1. ftrace system calls</a></li>
<li><a href="#trace-cmd">17.13.2.2. trace-cmd</a></li>
</ul>
</li>
<li><a href="#kprobes">17.13.3. Kprobes</a></li>
<li><a href="#count-boot-instructions">17.13.4. Count boot instructions</a></li>
</ul>
</li>
<li><a href="#linux-kernel-hardening">17.14. Linux kernel hardening</a>
<ul class="sectlevel3">
<li><a href="#config-fortify-source">17.14.1. CONFIG_FORTIFY_SOURCE</a></li>
<li><a href="#linux-security-modules">17.14.2. Linux security modules</a>
<ul class="sectlevel4">
<li><a href="#selinux">17.14.2.1. SELinux</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#user-mode-linux">17.15. User mode Linux</a></li>
<li><a href="#uio">17.16. UIO</a></li>
<li><a href="#linux-kernel-interactive-stuff">17.17. Linux kernel interactive stuff</a>
<ul class="sectlevel3">
<li><a href="#fbcon">17.17.1. Linux kernel console fun</a></li>
<li><a href="#linux-kernel-magic-keys">17.17.2. Linux kernel magic keys</a>
<ul class="sectlevel4">
<li><a href="#ctrl-alt-del">17.17.2.1. Ctrl Alt Del</a></li>
<li><a href="#sysrq">17.17.2.2. SysRq</a></li>
</ul>
</li>
<li><a href="#tty">17.17.3. TTY</a>
<ul class="sectlevel4">
<li><a href="#start-a-getty-from-outside-of-init">17.17.3.1. Start a getty from outside of init</a></li>
<li><a href="#console-kernel-boot-parameter">17.17.3.2. console kernel boot parameter</a></li>
</ul>
</li>
<li><a href="#config-logo">17.17.4. CONFIG_LOGO</a></li>
</ul>
</li>
<li><a href="#drm">17.18. DRM</a>
<ul class="sectlevel3">
<li><a href="#kmscube">17.18.1. kmscube</a></li>
<li><a href="#kmscon">17.18.2. kmscon</a></li>
<li><a href="#libdri2">17.18.3. libdri2</a></li>
</ul>
</li>
<li><a href="#linux-kernel-testing">17.19. Linux kernel testing</a>
<ul class="sectlevel3">
<li><a href="#linux-test-project">17.19.1. Linux Test Project</a></li>
<li><a href="#stress">17.19.2. stress</a></li>
</ul>
</li>
<li><a href="#linux-kernel-build-system">17.20. Linux kernel build system</a>
<ul class="sectlevel3">
<li><a href="#vmlinux-vs-bzimage-vs-zimage-vs-image">17.20.1. vmlinux vs bzImage vs zImage vs Image</a></li>
</ul>
</li>
<li><a href="#virtio">17.21. Virtio</a></li>
<li><a href="#kernel-modules">17.22. Kernel modules</a>
<ul class="sectlevel3">
<li><a href="#dump-regs">17.22.1. dump_regs</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#freebsd">18. FreeBSD</a></li>
<li><a href="#rtos">19. RTOS</a>
<ul class="sectlevel2">
<li><a href="#zephyr">19.1. Zephyr</a></li>
<li><a href="#arm-mbed">19.2. ARM Mbed</a></li>
</ul>
</li>
<li><a href="#xen">20. Xen</a></li>
<li><a href="#u-boot">21. U-Boot</a></li>
<li><a href="#emulators">22. Emulators</a></li>
<li><a href="#qemu">23. QEMU</a>
<ul class="sectlevel2">
<li><a href="#introduction-to-qemu">23.1. Introduction to QEMU</a></li>
<li><a href="#binary-translation">23.2. Binary translation</a></li>
<li><a href="#disk-persistency">23.3. Disk persistency</a>
<ul class="sectlevel3">
<li><a href="#gem5-disk-persistency">23.3.1. gem5 disk persistency</a></li>
</ul>
</li>
<li><a href="#gem5-qcow2">23.4. gem5 qcow2</a></li>
<li><a href="#snapshot">23.5. Snapshot</a>
<ul class="sectlevel3">
<li><a href="#snapshot-internals">23.5.1. Snapshot internals</a></li>
</ul>
</li>
<li><a href="#device-models">23.6. Device models</a>
<ul class="sectlevel3">
<li><a href="#pci">23.6.1. PCI</a>
<ul class="sectlevel4">
<li><a href="#qemu-edu">23.6.1.1. QEMU edu PCI device</a></li>
<li><a href="#manipulate-pci-registers-directly">23.6.1.2. Manipulate PCI registers directly</a></li>
<li><a href="#pciutils">23.6.1.3. pciutils</a></li>
<li><a href="#introduction-to-pci">23.6.1.4. Introduction to PCI</a></li>
<li><a href="#pci-bfd">23.6.1.5. PCI BFD</a></li>
<li><a href="#pci-bar">23.6.1.6. PCI BAR</a></li>
</ul>
</li>
<li><a href="#gpio">23.6.2. GPIO</a></li>
<li><a href="#leds">23.6.3. LEDs</a></li>
<li><a href="#gem5-educational-hardware-models">23.6.4. gem5 educational hardware models</a></li>
</ul>
</li>
<li><a href="#qemu-monitor">23.7. QEMU monitor</a>
<ul class="sectlevel3">
<li><a href="#qemu-monitor-from-guest">23.7.1. QEMU monitor from guest</a></li>
<li><a href="#qemu-monitor-from-gdb">23.7.2. QEMU monitor from GDB</a></li>
</ul>
</li>
<li><a href="#debug-the-emulator">23.8. Debug the emulator</a>
<ul class="sectlevel3">
<li><a href="#reverse-debug-the-emulator">23.8.1. Reverse debug the emulator</a></li>
<li><a href="#debug-gem5-python-scripts">23.8.2. Debug gem5 Python scripts</a></li>
</ul>
</li>
<li><a href="#tracing">23.9. Tracing</a>
<ul class="sectlevel3">
<li><a href="#qemu-d-tracing">23.9.1. QEMU -d tracing</a></li>
<li><a href="#qemu-trace-register-values">23.9.2. QEMU trace register values</a></li>
<li><a href="#qemu-trace-memory-accesses">23.9.3. QEMU trace memory accesses</a></li>
<li><a href="#trace-source-lines">23.9.4. Trace source lines</a></li>
<li><a href="#qemu-record-and-replay">23.9.5. QEMU record and replay</a>
<ul class="sectlevel4">
<li><a href="#qemu-reverse-debugging">23.9.5.1. QEMU reverse debugging</a></li>
</ul>
</li>
<li><a href="#qemu-trace-multicore">23.9.6. QEMU trace multicore</a></li>
<li><a href="#qemu-get-guest-instruction-count">23.9.7. QEMU get guest instruction count</a></li>
<li><a href="#gem5-tracing">23.9.8. gem5 tracing</a>
<ul class="sectlevel4">
<li><a href="#gem5-trace-internals">23.9.8.1. gem5 trace internals</a></li>
<li><a href="#gem5-execall-trace-format">23.9.8.2. gem5 ExecAll trace format</a></li>
<li><a href="#gem5-registers-trace-format">23.9.8.3. gem5 Registers trace format</a></li>
<li><a href="#gem5-tarmac-traces">23.9.8.4. gem5 TARMAC traces</a></li>
<li><a href="#gem5-tracing-internals">23.9.8.5. gem5 tracing internals</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#qemu-gui-is-unresponsive">23.10. QEMU GUI is unresponsive</a></li>
</ul>
</li>
<li><a href="#gem5">24. gem5</a>
<ul class="sectlevel2">
<li><a href="#gem5-vs-qemu">24.1. gem5 vs QEMU</a></li>
<li><a href="#gem5-run-benchmark">24.2. gem5 run benchmark</a>
<ul class="sectlevel3">
<li><a href="#skip-extra-benchmark-instructions">24.2.1. Skip extra benchmark instructions</a></li>
</ul>
</li>
<li><a href="#gem5-system-parameters">24.3. gem5 system parameters</a>
<ul class="sectlevel3">
<li><a href="#number-of-cores">24.3.1. Number of cores</a>
<ul class="sectlevel4">
<li><a href="#qemu-user-mode-multithreading">24.3.1.1. QEMU user mode multithreading</a></li>
<li><a href="#gem5-arm-full-system-with-more-than-8-cores">24.3.1.2. gem5 ARM full system with more than 8 cores</a></li>
</ul>
</li>
<li><a href="#gem5-cache-size">24.3.2. gem5 cache size</a></li>
<li><a href="#gem5-dram-model">24.3.3. gem5 DRAM model</a>
<ul class="sectlevel4">
<li><a href="#gem5-memory-latency">24.3.3.1. gem5 memory latency</a></li>
<li><a href="#memory-size">24.3.3.2. Memory size</a></li>
<li><a href="#gem5-dram-setup">24.3.3.3. gem5 DRAM setup</a></li>
</ul>
</li>
<li><a href="#gem5-disk-and-network-latency">24.3.4. gem5 disk and network latency</a></li>
<li><a href="#gem5-clock-frequency">24.3.5. gem5 clock frequency</a></li>
</ul>
</li>
<li><a href="#gem5-kernel-command-line-parameters">24.4. gem5 kernel command line parameters</a></li>
<li><a href="#gem5-gdb">24.5. gem5 GDB step debug</a>
<ul class="sectlevel3">
<li><a href="#gem5-gdb-step-debug-kernel">24.5.1. gem5 GDB step debug kernel</a></li>
<li><a href="#gem5-gdb-step-debug-userland-process">24.5.2. gem5 GDB step debug userland process</a></li>
<li><a href="#gem5-gdb-step-debug-secondary-cores">24.5.3. gem5 GDB step debug secondary cores</a></li>
</ul>
</li>
<li><a href="#gem5-checkpoint">24.6. gem5 checkpoint</a>
<ul class="sectlevel3">
<li><a href="#gem5-checkpoint-userland-minimal-example">24.6.1. gem5 checkpoint userland minimal example</a></li>
<li><a href="#gem5-checkpoint-internals">24.6.2. gem5 checkpoint internals</a></li>
<li><a href="#gem5-restore-new-script">24.6.3. gem5 checkpoint restore and run a different script</a></li>
<li><a href="#gem5-restore-checkpoint-with-a-different-cpu">24.6.4. gem5 restore checkpoint with a different CPU</a>
<ul class="sectlevel4">
<li><a href="#gem5-fast-forward">24.6.4.1. gem5 fast forward</a></li>
</ul>
</li>
<li><a href="#gem5-checkpoint-upgrader">24.6.5. gem5 checkpoint upgrader</a></li>
</ul>
</li>
<li><a href="#pass-extra-options-to-gem5">24.7. Pass extra options to gem5</a></li>
<li><a href="#m5ops">24.8. m5ops</a>
<ul class="sectlevel3">
<li><a href="#gem5-m5-executable">24.8.1. gem5 m5 executable</a>
<ul class="sectlevel4">
<li><a href="#m5-exit">24.8.1.1. m5 exit</a></li>
<li><a href="#m5-dumpstats">24.8.1.2. m5 dumpstats</a></li>
<li><a href="#m5-fail">24.8.1.3. m5 fail</a></li>
<li><a href="#m5-writefile">24.8.1.4. m5 writefile</a></li>
<li><a href="#m5-readfile">24.8.1.5. m5 readfile</a></li>
<li><a href="#m5-initparam">24.8.1.6. m5 initparam</a></li>
<li><a href="#m5-execfile">24.8.1.7. m5 execfile</a></li>
</ul>
</li>
<li><a href="#m5ops-instructions">24.8.2. m5ops instructions</a>
<ul class="sectlevel4">
<li><a href="#m5ops-magic-addresses">24.8.2.1. m5ops magic addresses</a></li>
<li><a href="#m5ops-instructions-interface">24.8.2.2. m5ops instructions interface</a></li>
<li><a href="#m5op-annotations">24.8.2.3. m5op annotations</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-arm-linux-kernel-patches">24.9. gem5 arm Linux kernel patches</a>
<ul class="sectlevel3">
<li><a href="#gem5-arm-linux-kernel-patches-boot-speedup">24.9.1. gem5 arm Linux kernel patches boot speedup</a></li>
</ul>
</li>
<li><a href="#m5out-directory">24.10. m5out directory</a>
<ul class="sectlevel3">
<li><a href="#gem5-m5out-system-terminal-file">24.10.1. gem5 m5out/system.terminal file</a></li>
<li><a href="#gem5-m5out-system-dmesg-file">24.10.2. gem5 <code>m5out/system.workload.dmesg</code> file</a></li>
<li><a href="#gem5-m5out-stats-txt-file">24.10.3. gem5 m5out/stats.txt file</a>
<ul class="sectlevel4">
<li><a href="#gem5-hdf5-statistics">24.10.3.1. gem5 HDF5 statistics</a></li>
<li><a href="#gem5-only-dump-selected-stats">24.10.3.2. gem5 only dump selected stats</a></li>
<li><a href="#meaning-of-each-gem5-stat">24.10.3.3. Meaning of each gem5 stat</a></li>
<li><a href="#gem5-stats-internals">24.10.3.4. gem5 stats internals</a></li>
</ul>
</li>
<li><a href="#gem5-config-ini">24.10.4. gem5 config.ini</a>
<ul class="sectlevel4">
<li><a href="#gem5-config-dot">24.10.4.1. gem5 config.dot</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#m5term">24.11. m5term</a></li>
<li><a href="#gem5-python-scripts-without-rebuild">24.12. gem5 Python scripts without rebuild</a></li>
<li><a href="#gem5-fs-biglittle">24.13. gem5 fs_bigLITTLE</a></li>
<li><a href="#gem5-in-tree-tests">24.14. gem5 in-tree tests</a>
<ul class="sectlevel3">
<li><a href="#gem5-unit-tests">24.14.1. gem5 unit tests</a></li>
<li><a href="#gem5-regression-tests">24.14.2. gem5 regression tests</a></li>
</ul>
</li>
<li><a href="#gem5-simulate-limit-reached">24.15. gem5 simulate() limit reached</a></li>
<li><a href="#gem5-build-options">24.16. gem5 build options</a>
<ul class="sectlevel3">
<li><a href="#gem5-debug-build">24.16.1. gem5 debug build</a></li>
<li><a href="#gem5-fast-build">24.16.2. gem5 fast build</a></li>
<li><a href="#gem5-prof-and-perf-builds">24.16.3. gem5 prof and perf builds</a></li>
<li><a href="#gem5-clang-build">24.16.4. gem5 clang build</a></li>
<li><a href="#gem5-sanitation-build">24.16.5. gem5 sanitation build</a></li>
<li><a href="#gem5-ruby-build">24.16.6. gem5 Ruby build</a>
<ul class="sectlevel4">
<li><a href="#gem5-ruby-mi-example-protocol">24.16.6.1. gem5 Ruby MI_example protocol</a></li>
<li><a href="#gem5-crossbar-interconnect">24.16.6.2. gem5 crossbar interconnect</a></li>
</ul>
</li>
<li><a href="#gem5-python-3-build">24.16.7. gem5 Python 3 build</a></li>
</ul>
</li>
<li><a href="#gem5-cpu-types">24.17. gem5 CPU types</a>
<ul class="sectlevel3">
<li><a href="#list-of-gem5-cpu-types">24.17.1. List of gem5 CPU types</a>
<ul class="sectlevel4">
<li><a href="#gem5-basesimplecpu">24.17.1.1. gem5 <code>BaseSimpleCPU</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-atomicsimplecpu">24.17.1.1.1. gem5 <code>AtomicSimpleCPU</code></a></li>
<li><a href="#gem5-timingsimplecpu">24.17.1.1.2. gem5 <code>TimingSimpleCPU</code></a></li>
</ul>
</li>
<li><a href="#gem5-minorcpu">24.17.1.2. gem5 MinorCPU</a></li>
<li><a href="#gem5-derivo3cpu">24.17.1.3. gem5 <code>DerivO3CPU</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-derivo3cpu-pipeline-stages">24.17.1.3.1. gem5 <code>DerivO3CPU</code> pipeline stages</a></li>
<li><a href="#gem5-util-o3-pipeview-py-o3-pipeline-viewer">24.17.1.3.2. gem5 util/o3-pipeview.py O3 pipeline viewer</a></li>
<li><a href="#gem5-konata-o3-pipeline-viewer">24.17.1.3.3. gem5 Konata O3 pipeline viewer</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-arm-rsk">24.17.2. gem5 ARM RSK</a></li>
</ul>
</li>
<li><a href="#gem5-arm-platforms">24.18. gem5 ARM platforms</a></li>
<li><a href="#gem5-upstream-images">24.19. gem5 upstream images</a></li>
<li><a href="#gem5-bootloaders">24.20. gem5 bootloaders</a></li>
<li><a href="#gem5-memory-system">24.21. gem5 memory system</a>
<ul class="sectlevel3">
<li><a href="#gem5-port-system">24.21.1. gem5 port system</a>
<ul class="sectlevel4">
<li><a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">24.21.1.1. gem5 functional vs atomic vs timing memory requests</a>
<ul class="sectlevel5">
<li><a href="#gem5-functional-requests">24.21.1.1.1. gem5 functional requests</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-packet-vs-request">24.21.2. gem5 <code>Packet</code> vs <code>Request</code></a>
<ul class="sectlevel4">
<li><a href="#gem5-packet">24.21.2.1. gem5 <code>Packet</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-memcmd">24.21.2.1.1. gem5 <code>MemCmd</code></a></li>
</ul>
</li>
<li><a href="#gem5-request">24.21.2.2. gem5 <code>Request</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-request-in-atomicsimplecpu">24.21.2.2.1. gem5 <code>Request</code> in <code>AtomicSimpleCPU</code></a></li>
<li><a href="#gem5-request-in-timingsimplecpu">24.21.2.2.2. gem5 <code>Request</code> in <code>TimingSimpleCPU</code></a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-mshr">24.21.3. gem5 <code>MSHR</code></a></li>
<li><a href="#gem5-commmonitor">24.21.4. gem5 <code>CommMonitor</code></a></li>
<li><a href="#gem5-simplememory">24.21.5. gem5 <code>SimpleMemory</code></a></li>
</ul>
</li>
<li><a href="#gem5-internals">24.22. gem5 internals</a>
<ul class="sectlevel3">
<li><a href="#gem5-eclipse-configuration">24.22.1. gem5 Eclipse configuration</a></li>
<li><a href="#gem5-python-c-interaction">24.22.2. gem5 Python C++ interaction</a></li>
<li><a href="#gem5-entry-point">24.22.3. gem5 entry point</a>
<ul class="sectlevel4">
<li><a href="#gem5-m5-objects-module">24.22.3.1. gem5 <code>m5.objects</code> module</a></li>
</ul>
</li>
<li><a href="#gem5-event-queue">24.22.4. gem5 event queue</a>
<ul class="sectlevel4">
<li><a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis">24.22.4.1. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis</a>
<ul class="sectlevel5">
<li><a href="#atomicsimplecpu-initial-events">24.22.4.1.1. AtomicSimpleCPU initial events</a></li>
<li><a href="#atomicsimplecpu-tick-reschedule-timing">24.22.4.1.2. AtomicSimpleCPU tick reschedule timing</a></li>
<li><a href="#atomicsimplecpu-memory-access">24.22.4.1.3. AtomicSimpleCPU memory access</a></li>
<li><a href="#gem5-se-py-page-translation">24.22.4.1.4. gem5 se.py page translation</a></li>
</ul>
</li>
<li><a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">24.22.4.2. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a>
<ul class="sectlevel5">
<li><a href="#timingsimplecpu-analysis-0">24.22.4.2.1. TimingSimpleCPU analysis #0</a></li>
<li><a href="#timingsimplecpu-analysis-1">24.22.4.2.2. TimingSimpleCPU analysis #1</a></li>
<li><a href="#timingsimplecpu-analysis-2">24.22.4.2.3. TimingSimpleCPU analysis #2</a></li>
<li><a href="#timingsimplecpu-analysis-3-and-4">24.22.4.2.4. TimingSimpleCPU analysis #3 and #4</a></li>
<li><a href="#timingsimplecpu-analysis-5">24.22.4.2.5. TimingSimpleCPU analysis #5</a></li>
<li><a href="#timingsimplecpu-analysis-6">24.22.4.2.6. TimingSimpleCPU analysis #6</a></li>
<li><a href="#timingsimplecpu-analysis-7">24.22.4.2.7. TimingSimpleCPU analysis #7</a></li>
<li><a href="#timingsimplecpu-analysis-8">24.22.4.2.8. TimingSimpleCPU analysis #8</a></li>
<li><a href="#timingsimplecpu-analysis-9">24.22.4.2.9. TimingSimpleCPU analysis #9</a></li>
<li><a href="#timingsimplecpu-analysis-10">24.22.4.2.10. TimingSimpleCPU analysis #10</a></li>
<li><a href="#timingsimplecpu-analysis-11">24.22.4.2.11. TimingSimpleCPU analysis #11</a></li>
<li><a href="#timingsimplecpu-analysis-12">24.22.4.2.12. TimingSimpleCPU analysis #12</a></li>
<li><a href="#timingsimplecpu-analysis-13">24.22.4.2.13. TimingSimpleCPU analysis #13</a></li>
<li><a href="#timingsimplecpu-analysis-14">24.22.4.2.14. TimingSimpleCPU analysis #14</a></li>
<li><a href="#timingsimplecpu-analysis-15">24.22.4.2.15. TimingSimpleCPU analysis #15</a></li>
<li><a href="#timingsimplecpu-analysis-16">24.22.4.2.16. TimingSimpleCPU analysis #16</a></li>
<li><a href="#timingsimplecpu-analysis-17">24.22.4.2.17. TimingSimpleCPU analysis #17</a></li>
<li><a href="#timingsimplecpu-analysis-18">24.22.4.2.18. TimingSimpleCPU analysis #18</a></li>
<li><a href="#timingsimplecpu-analysis-19">24.22.4.2.19. TimingSimpleCPU analysis #19</a></li>
<li><a href="#timingsimplecpu-analysis-20">24.22.4.2.20. TimingSimpleCPU analysis #20</a></li>
<li><a href="#timingsimplecpu-analysis-21">24.22.4.2.21. TimingSimpleCPU analysis #21</a></li>
<li><a href="#timingsimplecpu-analysis-22">24.22.4.2.22. TimingSimpleCPU analysis #22</a></li>
<li><a href="#timingsimplecpu-analysis-23">24.22.4.2.23. TimingSimpleCPU analysis #23</a></li>
<li><a href="#timingsimplecpu-analysis-24">24.22.4.2.24. TimingSimpleCPU analysis #24</a></li>
<li><a href="#timingsimplecpu-analysis-25">24.22.4.2.25. TimingSimpleCPU analysis #25</a></li>
<li><a href="#timingsimplecpu-analysis-26">24.22.4.2.26. TimingSimpleCPU analysis #26</a></li>
<li><a href="#timingsimplecpu-analysis-27">24.22.4.2.27. TimingSimpleCPU analysis #27</a></li>
<li><a href="#timingsimplecpu-analysis-28">24.22.4.2.28. TimingSimpleCPU analysis #28</a></li>
<li><a href="#timingsimplecpu-analysis-29">24.22.4.2.29. TimingSimpleCPU analysis #29</a></li>
<li><a href="#timingsimplecpu-analysis-ldr-stall">24.22.4.2.30. TimingSimpleCPU analysis: LDR stall</a></li>
</ul>
</li>
<li><a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">24.22.4.3. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a>
<ul class="sectlevel5">
<li><a href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5">24.22.4.3.1. What is the coherency protocol implemented by the classic cache system in gem5?</a></li>
</ul>
</li>
<li><a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">24.22.4.4. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></li>
<li><a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">24.22.4.5. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></li>
<li><a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus-and-ruby">24.22.4.6. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs and Ruby</a></li>
<li><a href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis">24.22.4.7. gem5 event queue MinorCPU syscall emulation freestanding example analysis</a>
<ul class="sectlevel5">
<li><a href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis-hazard">24.22.4.7.1. gem5 event queue MinorCPU syscall emulation freestanding example analysis: hazard</a></li>
</ul>
</li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis">24.22.4.8. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis</a>
<ul class="sectlevel5">
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless">24.22.4.8.1. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard">24.22.4.8.2. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard4">24.22.4.8.3. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard4</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall">24.22.4.8.4. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain">24.22.4.8.5. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_gain</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-hazard4">24.22.4.8.6. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_hazard4</a></li>
<li><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative">24.22.4.8.7. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gem5-instruction-definitions">24.22.5. gem5 instruction definitions</a>
<ul class="sectlevel4">
<li><a href="#gem5-execute-vs-initiateacc-vs-completeacc">24.22.5.1. gem5 <code>execute</code> vs <code>initiateAcc</code> vs <code>completeAcc</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-completeacc">24.22.5.1.1. gem5 <code>completeAcc</code></a></li>
</ul>
</li>
<li><a href="#gem5-microops">24.22.5.2. gem5 microops</a></li>
</ul>
</li>
<li><a href="#gem5-threadcontext-vs-threadstate-vs-execcontext-vs-process">24.22.6. gem5 <code>ThreadContext</code> vs <code>ThreadState</code> vs <code>ExecContext</code> vs <code>Process</code></a>
<ul class="sectlevel4">
<li><a href="#gem5-threadcontext">24.22.6.1. gem5 <code>ThreadContext</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-simplethread">24.22.6.1.1. gem5 <code>SimpleThread</code></a></li>
<li><a href="#gem5-o3threadcontext">24.22.6.1.2. gem5 <code>O3ThreadContext</code></a></li>
</ul>
</li>
<li><a href="#gem5-threadstate">24.22.6.2. gem5 <code>ThreadState</code></a></li>
<li><a href="#gem5-execcontext">24.22.6.3. gem5 <code>ExecContext</code></a>
<ul class="sectlevel5">
<li><a href="#gem5-execcontext-readintregoperand-register-resolution">24.22.6.3.1. gem5 <code>ExecContext::readIntRegOperand</code> register resolution</a></li>
</ul>
</li>
<li><a href="#gem5-process">24.22.6.4. gem5 <code>Process</code></a></li>
</ul>
</li>
<li><a href="#gem5-functional-units">24.22.7. gem5 functional units</a>
<ul class="sectlevel4">
<li><a href="#gem5-minorcpu-default-functional-units">24.22.7.1. gem5 <code>MinorCPU</code> default functional units</a></li>
<li><a href="#gem5-derivo3cpu-default-functional-units">24.22.7.2. gem5 DerivO3CPU default functional units</a></li>
</ul>
</li>
<li><a href="#gem5-code-generation">24.22.8. gem5 code generation</a>
<ul class="sectlevel4">
<li><a href="#gem5-the-isa">24.22.8.1. gem5 THE_ISA</a></li>
</ul>
</li>
<li><a href="#gem5-build-system">24.22.9. gem5 build system</a>
<ul class="sectlevel4">
<li><a href="#m5-override-py-source">24.22.9.1. M5_OVERRIDE_PY_SOURCE</a></li>
<li><a href="#gem5-build-broken-on-recent-compiler-version">24.22.9.2. gem5 build broken on recent compiler version</a></li>
<li><a href="#gem5-polymorphic-isa-includes">24.22.9.3. gem5 polymorphic ISA includes</a></li>
<li><a href="#why-are-all-c-symlinked-into-the-gem5-build-dir">24.22.9.4. Why are all C++ symlinked into the gem5 build dir?</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#gensim">25. Gensim</a></li>
<li><a href="#buildroot">26. Buildroot</a>
<ul class="sectlevel2">
<li><a href="#introduction-to-buildroot">26.1. Introduction to Buildroot</a></li>
<li><a href="#custom-buildroot-configs">26.2. Custom Buildroot configs</a>
<ul class="sectlevel3">
<li><a href="#enable-buildroot-compiler-optimizations">26.2.1. Enable Buildroot compiler optimizations</a></li>
</ul>
</li>
<li><a href="#find-buildroot-options-with-make-menuconfig">26.3. Find Buildroot options with make menuconfig</a></li>
<li><a href="#change-user">26.4. Change user</a>
<ul class="sectlevel3">
<li><a href="#login-as-a-non-root-user-without-password">26.4.1. Login as a non-root user without password</a></li>
</ul>
</li>
<li><a href="#add-new-files-to-the-buildroot-image">26.5. Add new files to the Buildroot image</a>
<ul class="sectlevel3">
<li><a href="#add-new-buildroot-packages">26.5.1. Add new Buildroot packages</a></li>
</ul>
</li>
<li><a href="#remove-buildroot-packages">26.6. Remove Buildroot packages</a></li>
<li><a href="#br2-target-rootfs-ext2-size">26.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a>
<ul class="sectlevel3">
<li><a href="#squashfs">26.7.1. SquashFS</a></li>
</ul>
</li>
<li><a href="#rpath">26.8. Buildroot rebuild is slow when the root filesystem is large</a></li>
<li><a href="#report-upstream-bugs">26.9. Report upstream bugs</a></li>
<li><a href="#libc-choice">26.10. libc choice</a></li>
<li><a href="#buildroot-hello-world">26.11. Buildroot hello world</a></li>
<li><a href="#update-the-buildroot-toolchain">26.12. Update the Buildroot toolchain</a>
<ul class="sectlevel3">
<li><a href="#update-gcc-gcc-supported-by-buildroot">26.12.1. Update GCC: GCC supported by Buildroot</a></li>
<li><a href="#update-gcc-gcc-not-supported-by-buildroot">26.12.2. Update GCC: GCC not supported by Buildroot</a></li>
</ul>
</li>
<li><a href="#buildroot-vanilla-kernel">26.13. Buildroot vanilla kernel</a></li>
</ul>
</li>
<li><a href="#userland-content">27. Userland content</a>
<ul class="sectlevel2">
<li><a href="#build-userland">27.1. build-userland</a></li>
<li><a href="#c">27.2. C</a>
<ul class="sectlevel3">
<li><a href="#malloc">27.2.1. malloc</a>
<ul class="sectlevel4">
<li><a href="#malloc-implementation">27.2.1.1. malloc implementation</a></li>
<li><a href="#malloc-maximum-size">27.2.1.2. malloc maximum size</a>
<ul class="sectlevel5">
<li><a href="#linux-out-of-memory-killer">27.2.1.2.1. Linux out-of-memory killer</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#c-multithreading">27.2.2. C multithreading</a>
<ul class="sectlevel4">
<li><a href="#atomic-c">27.2.2.1. atomic.c</a></li>
</ul>
</li>
<li><a href="#gcc-c-extensions">27.2.3. GCC C extensions</a>
<ul class="sectlevel4">
<li><a href="#c-empty-struct">27.2.3.1. C empty struct</a></li>
<li><a href="#openmp">27.2.3.2. OpenMP</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#cpp">27.3. C++</a>
<ul class="sectlevel3">
<li><a href="#cpp-classes">27.3.1. C++ classes</a>
<ul class="sectlevel4">
<li><a href="#cpp-constructor">27.3.1.1. C++ constructor</a>
<ul class="sectlevel5">
<li><a href="#cpp-rule-of-five">27.3.1.1.1. C++ rule of five</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#cpp-standards">27.3.2. C++ standards</a></li>
<li><a href="#cpp-initialization-types">27.3.3. C++ initialization types</a></li>
<li><a href="#cpp-multithreading">27.3.4. C++ multithreading</a>
<ul class="sectlevel4">
<li><a href="#atomic-cpp">27.3.4.1. atomic.cpp</a>
<ul class="sectlevel5">
<li><a href="#detailed-gem5-analysis-of-how-data-races-happen">27.3.4.1.1. Detailed gem5 analysis of how data races happen</a></li>
</ul>
</li>
<li><a href="#cpp-memory-order">27.3.4.2. C++ std::memory_order</a></li>
<li><a href="#cpp-parallel-algorithms">27.3.4.3. C++ parallel algorithms</a></li>
<li><a href="#cpp17">27.3.4.4. C++17 N4659 standards draft</a></li>
</ul>
</li>
<li><a href="#cpp-templates">27.3.5. C++ templates</a>
<ul class="sectlevel4">
<li><a href="#sfinae">27.3.5.1. SFINAE</a></li>
</ul>
</li>
<li><a href="#cpp-type-casting">27.3.6. C++ type casting</a></li>
<li><a href="#cpp-compile-time-magic">27.3.7. C++ compile time magic</a>
<ul class="sectlevel4">
<li><a href="#cpp-decltype">27.3.7.1. C++ <code>decltype</code></a></li>
</ul>
</li>
<li><a href="#cpp-concepts">27.3.8. C++ concepts</a>
<ul class="sectlevel4">
<li><a href="#cpp-iterators">27.3.8.1. C++ iterators</a></li>
</ul>
</li>
<li><a href="#cpp-third-party-libraries">27.3.9. C++ third-party libraries</a>
<ul class="sectlevel4">
<li><a href="#boost">27.3.9.1. Boost</a></li>
<li><a href="#googletest">27.3.9.2. GoogleTest</a></li>
<li><a href="#hdf5">27.3.9.3. HDF5</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#posix">27.4. POSIX</a>
<ul class="sectlevel3">
<li><a href="#environment-variables">27.4.1. Environment variables</a></li>
<li><a href="#unistd-h">27.4.2. unistd.h</a></li>
<li><a href="#fork">27.4.3. fork</a>
<ul class="sectlevel4">
<li><a href="#getpid">27.4.3.1. getpid</a></li>
<li><a href="#fork-bomb">27.4.3.2. Fork bomb</a></li>
</ul>
</li>
<li><a href="#pthreads">27.4.4. pthreads</a>
<ul class="sectlevel4">
<li><a href="#pthread-mutex">27.4.4.1. pthread_mutex</a></li>
</ul>
</li>
<li><a href="#sysconf">27.4.5. sysconf</a></li>
<li><a href="#mmap-2">27.4.6. mmap</a>
<ul class="sectlevel4">
<li><a href="#mmap-map-anonymous">27.4.6.1. mmap MAP_ANONYMOUS</a></li>
<li><a href="#mmap-file">27.4.6.2. mmap file</a></li>
<li><a href="#brk">27.4.6.3. brk</a></li>
</ul>
</li>
<li><a href="#socket">27.4.7. socket</a></li>
</ul>
</li>
<li><a href="#userland-multithreading">27.5. Userland multithreading</a></li>
<li><a href="#c-debugging">27.6. C debugging</a>
<ul class="sectlevel3">
<li><a href="#stack-smashing">27.6.1. Stack smashing</a></li>
<li><a href="#memory-leaks">27.6.2. Memory leaks</a></li>
<li><a href="#profiling-userland-programs">27.6.3. Profiling userland programs</a></li>
</ul>
</li>
<li><a href="#interpreted-languages">27.7. Interpreted languages</a>
<ul class="sectlevel3">
<li><a href="#python">27.7.1. Python</a>
<ul class="sectlevel4">
<li><a href="#python-standard-library">27.7.1.1. Python standard library</a>
<ul class="sectlevel5">
<li><a href="#python-unittest">27.7.1.1.1. Python unittest</a></li>
<li><a href="#python-relative-imports">27.7.1.1.2. Python relative imports</a></li>
</ul>
</li>
<li><a href="#build-and-install-the-interpreter">27.7.1.2. Build and install the interpreter</a></li>
<li><a href="#python-gem5-user-mode-simulation">27.7.1.3. Python gem5 user mode simulation</a></li>
<li><a href="#embedding-python-in-another-application">27.7.1.4. Embedding Python in another application</a></li>
<li><a href="#pybind11">27.7.1.5. pybind11</a></li>
</ul>
</li>
<li><a href="#node-js">27.7.2. Node.js</a>
<ul class="sectlevel4">
<li><a href="#node-js-step-debugging">27.7.2.1. Node.js step debugging</a></li>
<li><a href="#npm">27.7.2.2. NPM</a>
<ul class="sectlevel5">
<li><a href="#npm-data-files">27.7.2.2.1. NPM data-files</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#java">27.7.3. Java</a></li>
</ul>
</li>
<li><a href="#algorithms">27.8. Algorithms</a>
<ul class="sectlevel3">
<li><a href="#bst-vs-heap-vs-hashmap">27.8.1. BST vs heap vs hashmap</a></li>
<li><a href="#blas">27.8.2. BLAS</a></li>
<li><a href="#eigen">27.8.3. Eigen</a></li>
</ul>
</li>
<li><a href="#benchmarks">27.9. Benchmarks</a>
<ul class="sectlevel3">
<li><a href="#microbenchmarks">27.9.1. Microbenchmarks</a></li>
</ul>
</li>
<li><a href="#userland-libs-directory">27.10. userland/libs directory</a></li>
<li><a href="#userland-content-filename-conventions">27.11. Userland content filename conventions</a></li>
<li><a href="#userland-content-bibliography">27.12. Userland content bibliography</a></li>
</ul>
</li>
<li><a href="#userland-assembly">28. Userland assembly</a>
<ul class="sectlevel2">
<li><a href="#assembly-registers">28.1. Assembly registers</a>
<ul class="sectlevel3">
<li><a href="#armv8-aarch64-x31-register">28.1.1. ARMv8 aarch64 x31 register</a></li>
</ul>
</li>
<li><a href="#floating-point-assembly">28.2. Floating point assembly</a></li>
<li><a href="#simd-assembly">28.3. SIMD assembly</a>
<ul class="sectlevel3">
<li><a href="#fma-instruction">28.3.1. FMA instruction</a></li>
</ul>
</li>
<li><a href="#user-vs-system-assembly">28.4. User vs system assembly</a></li>
<li><a href="#userland-assembly-c-standard-library">28.5. Userland assembly C standard library</a>
<ul class="sectlevel3">
<li><a href="#freestanding-programs">28.5.1. Freestanding programs</a>
<ul class="sectlevel4">
<li><a href="#nostartfiles-programs">28.5.1.1. nostartfiles programs</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#gcc-inline-assembly">28.6. GCC inline assembly</a>
<ul class="sectlevel3">
<li><a href="#gcc-inline-assembly-register-variables">28.6.1. GCC inline assembly register variables</a></li>
<li><a href="#gcc-inline-assembly-scratch-registers">28.6.2. GCC inline assembly scratch registers</a></li>
<li><a href="#gcc-inline-assembly-early-clobbers">28.6.3. GCC inline assembly early-clobbers</a></li>
<li><a href="#gcc-inline-assembly-floating-point-arm">28.6.4. GCC inline assembly floating point ARM</a></li>
<li><a href="#gcc-intrinsics">28.6.5. GCC intrinsics</a>
<ul class="sectlevel4">
<li><a href="#gcc-x86-intrinsics">28.6.5.1. GCC x86 intrinsics</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#linux-system-calls">28.7. Linux system calls</a>
<ul class="sectlevel3">
<li><a href="#futex-system-call">28.7.1. futex system call</a>
<ul class="sectlevel4">
<li><a href="#userland-mutex-implementation">28.7.1.1. Userland mutex implementation</a></li>
</ul>
</li>
<li><a href="#getcpu">28.7.2. <code>getcpu</code> system call and the <code>sched_getaffinity</code> glibc wrapper</a></li>
<li><a href="#perf-event-open">28.7.3. <code>perf_event_open</code> system call</a></li>
</ul>
</li>
<li><a href="#linux-calling-conventions">28.8. Linux calling conventions</a>
<ul class="sectlevel3">
<li><a href="#x86-64-calling-convention">28.8.1. x86_64 calling convention</a></li>
<li><a href="#arm-calling-convention">28.8.2. ARM calling convention</a></li>
</ul>
</li>
<li><a href="#gnu-gas-assembler">28.9. GNU GAS assembler</a>
<ul class="sectlevel3">
<li><a href="#gnu-gas-assembler-comments">28.9.1. GNU GAS assembler comments</a></li>
<li><a href="#gnu-gas-assembler-immediates">28.9.2. GNU GAS assembler immediates</a></li>
<li><a href="#gnu-gas-assembler-data-sizes">28.9.3. GNU GAS assembler data sizes</a>
<ul class="sectlevel4">
<li><a href="#gnu-gas-assembler-arm-specifics">28.9.3.1. GNU GAS assembler ARM specifics</a>
<ul class="sectlevel5">
<li><a href="#gnu-gas-assembler-arm-unified-syntax">28.9.3.1.1. GNU GAS assembler ARM unified syntax</a></li>
</ul>
</li>
<li><a href="#gnu-gas-assembler-arm-n-and-w-suffixes">28.9.3.2. GNU GAS assembler ARM .n and .w suffixes</a></li>
</ul>
</li>
<li><a href="#gnu-gas-assembler-char-literals">28.9.4. GNU GAS assembler char literals</a></li>
</ul>
</li>
<li><a href="#nop-instructions">28.10. NOP instructions</a></li>
</ul>
</li>
<li><a href="#x86-userland-assembly">29. x86 userland assembly</a>
<ul class="sectlevel2">
<li><a href="#x86-registers">29.1. x86 registers</a>
<ul class="sectlevel3">
<li><a href="#x86-flags-registers">29.1.1. x86 FLAGS registers</a></li>
</ul>
</li>
<li><a href="#x86-addressing-modes">29.2. x86 addressing modes</a></li>
<li><a href="#x86-data-transfer-instructions">29.3. x86 data transfer instructions</a>
<ul class="sectlevel3">
<li><a href="#x86-exchange-instructions">29.3.1. x86 exchange instructions</a>
<ul class="sectlevel4">
<li><a href="#x86-cmpxchg-instruction">29.3.1.1. x86 CMPXCHG instruction</a></li>
</ul>
</li>
<li><a href="#x86-push-and-pop-instructions">29.3.2. x86 PUSH and POP instructions</a></li>
<li><a href="#x86-cqto-and-cltq-instructions">29.3.3. x86 CQTO and CLTQ instructions</a></li>
<li><a href="#x86-cmovcc-instructions">29.3.4. x86 CMOVcc instructions</a></li>
</ul>
</li>
<li><a href="#x86-binary-arithmetic-instructions">29.4. x86 binary arithmetic instructions</a></li>
<li><a href="#x86-logical-instructions">29.5. x86 logical instructions</a></li>
<li><a href="#x86-shift-and-rotate-instructions">29.6. x86 shift and rotate instructions</a></li>
<li><a href="#x86-bit-and-byte-instructions">29.7. x86 bit and byte instructions</a></li>
<li><a href="#x86-control-transfer-instructions">29.8. x86 control transfer instructions</a>
<ul class="sectlevel3">
<li><a href="#x86-jcc-instructions">29.8.1. x86 Jcc instructions</a></li>
<li><a href="#x86-loop-instruction">29.8.2. x86 LOOP instruction</a></li>
<li><a href="#x86-string-instructions">29.8.3. x86 string instructions</a>
<ul class="sectlevel4">
<li><a href="#x86-rep-prefix">29.8.3.1. x86 REP prefix</a></li>
</ul>
</li>
<li><a href="#x86-enter-and-leave-instructions">29.8.4. x86 ENTER and LEAVE instructions</a></li>
</ul>
</li>
<li><a href="#x86-miscellaneous-instructions">29.9. x86 miscellaneous instructions</a></li>
<li><a href="#x86-random-number-generator-instructions">29.10. x86 random number generator instructions</a>
<ul class="sectlevel3">
<li><a href="#x86-cpuid-instruction">29.10.1. x86 CPUID instruction</a></li>
</ul>
</li>
<li><a href="#x86-x87-fpu-instructions">29.11. x86 x87 FPU instructions</a>
<ul class="sectlevel3">
<li><a href="#x86-x87-fpu-vs-simd">29.11.1. x86 x87 FPU vs SIMD</a></li>
</ul>
</li>
<li><a href="#x86-simd">29.12. x86 SIMD</a>
<ul class="sectlevel3">
<li><a href="#x86-sse-instructions">29.12.1. x86 SSE instructions</a>
<ul class="sectlevel4">
<li><a href="#x86-sse-data-transfer-instructions">29.12.1.1. x86 SSE data transfer instructions</a></li>
<li><a href="#x86-sse-packed-arithmetic-instructions">29.12.1.2. x86 SSE packed arithmetic instructions</a></li>
<li><a href="#x86-sse-conversion-instructions">29.12.1.3. x86 SSE conversion instructions</a></li>
</ul>
</li>
<li><a href="#x86-sse2-instructions">29.12.2. x86 SSE2 instructions</a>
<ul class="sectlevel4">
<li><a href="#x86-paddq-instruction">29.12.2.1. x86 PADDQ instruction</a></li>
</ul>
</li>
<li><a href="#x86-fma">29.12.3. x86 fused multiply add (FMA)</a></li>
</ul>
</li>
<li><a href="#x86-system-instructions">29.13. x86 system instructions</a>
<ul class="sectlevel3">
<li><a href="#x86-rdtsc-instruction">29.13.1. x86 RDTSC instruction</a>
<ul class="sectlevel4">
<li><a href="#x86-rdtscp-instruction">29.13.1.1. x86 RDTSCP instruction</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#x86-thread-synchronization-primitives">29.14. x86 thread synchronization primitives</a>
<ul class="sectlevel3">
<li><a href="#x86-lock-prefix">29.14.1. x86 LOCK prefix</a></li>
</ul>
</li>
<li><a href="#x86-assembly-bibliography">29.15. x86 assembly bibliography</a>
<ul class="sectlevel3">
<li><a href="#x86-official-bibliography">29.15.1. x86 official bibliography</a>
<ul class="sectlevel4">
<li><a href="#intel-manual">29.15.1.1. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals</a>
<ul class="sectlevel5">
<li><a href="#intel-manual-1">29.15.1.1.1. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a></li>
<li><a href="#intel-manual-2">29.15.1.1.2. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 2</a></li>
<li><a href="#intel-manual-3">29.15.1.1.3. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 3</a></li>
<li><a href="#intel-manual-4">29.15.1.1.4. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 4</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-userland-assembly">30. ARM userland assembly</a>
<ul class="sectlevel2">
<li><a href="#introduction-to-the-arm-architecture">30.1. Introduction to the ARM architecture</a>
<ul class="sectlevel3">
<li><a href="#armv8-vs-armv7-vs-aarch64-vs-aarch32">30.1.1. ARMv8 vs ARMv7 vs AArch64 vs AArch32</a>
<ul class="sectlevel4">
<li><a href="#aarch32">30.1.1.1. AArch32</a></li>
<li><a href="#aarch32-vs-aarch64">30.1.1.2. AArch32 vs AArch64</a></li>
</ul>
</li>
<li><a href="#free-arm-implementations">30.1.2. Free ARM implementations</a></li>
<li><a href="#arm-instruction-encodings">30.1.3. ARM instruction encodings</a>
<ul class="sectlevel4">
<li><a href="#arm-thumb-encoding">30.1.3.1. ARM Thumb encoding</a></li>
<li><a href="#arm-big-endian-mode">30.1.3.2. ARM big endian mode</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-branch-instructions">30.2. ARM branch instructions</a>
<ul class="sectlevel3">
<li><a href="#arm-b-instruction">30.2.1. ARM B instruction</a></li>
<li><a href="#arm-beq-instruction">30.2.2. ARM BEQ instruction</a></li>
<li><a href="#arm-bl-instruction">30.2.3. ARM BL instruction</a>
<ul class="sectlevel4">
<li><a href="#arm-bx-instruction">30.2.3.1. ARM BX instruction</a></li>
<li><a href="#armv8-aarch64-ret-instruction">30.2.3.2. ARMv8 aarch64 ret instruction</a></li>
</ul>
</li>
<li><a href="#arm-cbz-instruction">30.2.4. ARM CBZ instruction</a></li>
<li><a href="#arm-conditional-execution">30.2.5. ARM conditional execution</a></li>
</ul>
</li>
<li><a href="#arm-load-and-store-instructions">30.3. ARM load and store instructions</a>
<ul class="sectlevel3">
<li><a href="#arm-ldr-instruction">30.3.1. ARM LDR instruction</a>
<ul class="sectlevel4">
<li><a href="#arm-ldr-pseudo-instruction">30.3.1.1. ARM LDR pseudo-instruction</a></li>
<li><a href="#arm-addressing-modes">30.3.1.2. ARM addressing modes</a>
<ul class="sectlevel5">
<li><a href="#arm-loop-over-array">30.3.1.2.1. ARM loop over array</a></li>
</ul>
</li>
<li><a href="#arm-ldrh-and-ldrb-instructions">30.3.1.3. ARM LDRH and LDRB instructions</a></li>
</ul>
</li>
<li><a href="#arm-str-instruction">30.3.2. ARM STR instruction</a>
<ul class="sectlevel4">
<li><a href="#armv8-aarch64-str-instruction">30.3.2.1. ARMv8 aarch64 STR instruction</a></li>
<li><a href="#armv8-aarch64-ldp-and-stp-instructions">30.3.2.2. ARMv8 aarch64 LDP and STP instructions</a>
<ul class="sectlevel5">
<li><a href="#armv8-aarch64-stack-alignment">30.3.2.2.1. ARMV8 aarch64 stack alignment</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-ldmia-instruction">30.3.3. ARM LDMIA instruction</a></li>
</ul>
</li>
<li><a href="#arm-data-processing-instructions">30.4. ARM data processing instructions</a>
<ul class="sectlevel3">
<li><a href="#arm-cset-instruction">30.4.1. ARM CSET instruction</a></li>
<li><a href="#arm-bitwise-instructions">30.4.2. ARM bitwise instructions</a>
<ul class="sectlevel4">
<li><a href="#arm-bic-instruction">30.4.2.1. ARM BIC instruction</a></li>
<li><a href="#arm-ubfm-instruction">30.4.2.2. ARM UBFM instruction</a>
<ul class="sectlevel5">
<li><a href="#arm-ubfx-instruction">30.4.2.2.1. ARM UBFX instruction</a></li>
</ul>
</li>
<li><a href="#arm-bfm-instruction">30.4.2.3. ARM BFM instruction</a>
<ul class="sectlevel5">
<li><a href="#arm-bfi-instruction">30.4.2.3.1. ARM BFI instruction</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-mov-instruction">30.4.3. ARM MOV instruction</a>
<ul class="sectlevel4">
<li><a href="#arm-movw-and-movt-instructions">30.4.3.1. ARM movw and movt instructions</a></li>
<li><a href="#armv8-aarch64-movk-instruction">30.4.3.2. ARMv8 aarch64 movk instruction</a></li>
<li><a href="#armv8-aarch64-movn-instruction">30.4.3.3. ARMv8 aarch64 movn instruction</a></li>
</ul>
</li>
<li><a href="#arm-data-processing-instruction-suffixes">30.4.4. ARM data processing instruction suffixes</a>
<ul class="sectlevel4">
<li><a href="#arm-shift-suffixes">30.4.4.1. ARM shift suffixes</a></li>
<li><a href="#arm-s-suffix">30.4.4.2. ARM S suffix</a></li>
</ul>
</li>
<li><a href="#arm-adr-instruction">30.4.5. ARM ADR instruction</a>
<ul class="sectlevel4">
<li><a href="#arm-adrl-instruction">30.4.5.1. ARM ADRL instruction</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-miscellaneous-instructions">30.5. ARM miscellaneous instructions</a>
<ul class="sectlevel3">
<li><a href="#arm-nop-instruction">30.5.1. ARM NOP instruction</a></li>
<li><a href="#arm-udf-instruction">30.5.2. ARM UDF instruction</a></li>
<li><a href="#arm-system-register-instructions">30.5.3. ARM system register instructions</a>
<ul class="sectlevel4">
<li><a href="#arm-system-register-encodings">30.5.3.1. ARM system register encodings</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-simd">30.6. ARM SIMD</a>
<ul class="sectlevel3">
<li><a href="#arm-vfp">30.6.1. ARM VFP</a>
<ul class="sectlevel4">
<li><a href="#arm-vfp-registers">30.6.1.1. ARM VFP registers</a></li>
<li><a href="#arm-vadd-instruction">30.6.1.2. ARM VADD instruction</a></li>
<li><a href="#arm-vcvt-instruction">30.6.1.3. ARM VCVT instruction</a>
<ul class="sectlevel5">
<li><a href="#arm-vcvtr-instruction">30.6.1.3.1. ARM VCVTR instruction</a></li>
<li><a href="#armv8-aarch32-vcvta-instruction">30.6.1.3.2. ARMv8 AArch32 VCVTA instruction</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#armv8-advanced-simd-and-floating-point-support">30.6.2. ARMv8 Advanced SIMD and floating-point support</a>
<ul class="sectlevel4">
<li><a href="#armv8-floating-point-availability">30.6.2.1. ARMv8 floating point availability</a></li>
<li><a href="#arm-neon">30.6.2.2. ARM NEON</a></li>
</ul>
</li>
<li><a href="#armv8-aarch64-floating-point-registers">30.6.3. ARMv8 AArch64 floating point registers</a>
<ul class="sectlevel4">
<li><a href="#armv8-aarch64-add-vector-instruction">30.6.3.1. ARMv8 aarch64 add vector instruction</a></li>
<li><a href="#armv8-aarch64-fadd-instruction">30.6.3.2. ARMv8 aarch64 FADD instruction</a>
<ul class="sectlevel5">
<li><a href="#arm-fadd-vs-vadd">30.6.3.2.1. ARM FADD vs VADD</a></li>
</ul>
</li>
<li><a href="#armv8-aarch64-ld2-instruction">30.6.3.3. ARMv8 aarch64 LD2 instruction</a></li>
</ul>
</li>
<li><a href="#arm-simd-bibliography">30.6.4. ARM SIMD bibliography</a></li>
<li><a href="#arm-sve">30.6.5. ARM SVE</a>
<ul class="sectlevel4">
<li><a href="#arm-sve-vaddl-instruction">30.6.5.1. ARM SVE VADDL instruction</a></li>
<li><a href="#change-arm-sve-vector-length-in-emulators">30.6.5.2. Change ARM SVE vector length in emulators</a></li>
<li><a href="#sve-bibliography">30.6.5.3. SVE bibliography</a>
<ul class="sectlevel5">
<li><a href="#sve-spec">30.6.5.3.1. SVE spec</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#arm-thread-synchronization-primitives">30.7. ARM thread synchronization primitives</a>
<ul class="sectlevel3">
<li><a href="#arm-ldxr-and-stxr-instructions">30.7.1. ARM LDXR and STXR instructions</a></li>
<li><a href="#arm-lse">30.7.2. ARM Large System Extensions (LSE)</a></li>
</ul>
</li>
<li><a href="#armv8-architecture-extensions">30.8. ARMv8 architecture extensions</a>
<ul class="sectlevel3">
<li><a href="#armv8-1-architecture-extension">30.8.1. ARMv8.1 architecture extension</a></li>
</ul>
</li>
<li><a href="#arm-pmu">30.9. ARM PMU</a>
<ul class="sectlevel3">
<li><a href="#arm-pmccntr-register">30.9.1. ARM PMCCNTR register</a></li>
</ul>
</li>
<li><a href="#arm-assembly-bibliography">30.10. ARM assembly bibliography</a>
<ul class="sectlevel3">
<li><a href="#arm-non-official-bibliography">30.10.1. ARM non-official bibliography</a></li>
<li><a href="#arm-official-bibliography">30.10.2. ARM official bibliography</a>
<ul class="sectlevel4">
<li><a href="#armarm7">30.10.2.1. ARMv7 architecture reference manual</a></li>
<li><a href="#armarm8">30.10.2.2. ARMv8 architecture reference manual</a></li>
<li><a href="#armarm8-db">30.10.2.3. ARMv8 architecture reference manual db</a></li>
<li><a href="#armarm8-fa">30.10.2.4. ARMv8 architecture reference manual db</a></li>
<li><a href="#armv8-programmers-guide">30.10.2.5. Programmer&#8217;s Guide for ARMv8-A</a></li>
<li><a href="#arm-a64-instruction-set-architecture-future-architecture-technologies-in-the-a-architecture-profile-documentation">30.10.2.6. Arm A64 Instruction Set Architecture: Future Architecture Technologies in the A architecture profile Documentation</a></li>
<li><a href="#arm-processor-documentation">30.10.2.7. ARM processor documentation</a>
<ul class="sectlevel5">
<li><a href="#arm-cortex15-trm">30.10.2.7.1. ARM Cortex-A15 MPCore Processor Technical Reference Manual r4p0</a></li>
</ul>
</li>
<li><a href="#arm-cortex-a77-trm">30.10.2.8. Arm Cortex‑A77 Technical Reference Manual r1p1</a></li>
<li><a href="#arm-cortex-a77-sog">30.10.2.9. Arm Cortex‑A77 Software Optimization Guide r1p1</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#elf">31. ELF</a></li>
<li><a href="#ieee-754">32. IEEE 754</a></li>
<li><a href="#baremetal">33. Baremetal</a>
<ul class="sectlevel2">
<li><a href="#baremetal-gdb-step-debug">33.1. Baremetal GDB step debug</a></li>
<li><a href="#baremetal-bootloaders">33.2. Baremetal bootloaders</a></li>
<li><a href="#baremetal-linker-script">33.3. Baremetal linker script</a></li>
<li><a href="#baremetal-command-line-arguments">33.4. Baremetal command line arguments</a>
<ul class="sectlevel3">
<li><a href="#gem5-baremetal-arm-cli-args">33.4.1. gem5 baremetal arm CLI args</a></li>
</ul>
</li>
<li><a href="#semihosting">33.5. Semihosting</a>
<ul class="sectlevel3">
<li><a href="#gem5-semihosting">33.5.1. gem5 semihosting</a></li>
</ul>
</li>
<li><a href="#gem5-baremetal-carriage-return">33.6. gem5 baremetal carriage return</a></li>
<li><a href="#baremetal-host-packaged-toolchain">33.7. Baremetal host packaged toolchain</a></li>
<li><a href="#baremetal-cpp">33.8. Baremetal C++</a></li>
<li><a href="#gdb-builtin-cpu-simulator">33.9. GDB builtin CPU simulator</a>
<ul class="sectlevel3">
<li><a href="#gdb-builtin-cpu-simulator-userland">33.9.1. GDB builtin CPU simulator userland</a></li>
</ul>
</li>
<li><a href="#arm-baremetal">33.10. ARM baremetal</a>
<ul class="sectlevel3">
<li><a href="#arm-exception-levels">33.10.1. ARM exception levels</a>
<ul class="sectlevel4">
<li><a href="#arm-change-exception-level">33.10.1.1. ARM change exception level</a></li>
<li><a href="#arm-sp0-vs-spx">33.10.1.2. ARM SP0 vs SPx</a></li>
</ul>
</li>
<li><a href="#arm-svc-instruction">33.10.2. ARM SVC instruction</a>
<ul class="sectlevel4">
<li><a href="#armv8-exception-vector-table-format">33.10.2.1. ARMv8 exception vector table format</a></li>
<li><a href="#arm-esr-register">33.10.2.2. ARM ESR register</a></li>
<li><a href="#arm-elr-register">33.10.2.3. ARM ELR register</a></li>
</ul>
</li>
<li><a href="#arm-baremetal-multicore">33.10.3. ARM baremetal multicore</a>
<ul class="sectlevel4">
<li><a href="#arm-wfe-and-sev-instructions">33.10.3.1. ARM WFE and SEV instructions</a>
<ul class="sectlevel5">
<li><a href="#arm-wfe-global-monitor-events">33.10.3.1.1. ARM WFE global monitor events</a></li>
<li><a href="#wfe-from-userland">33.10.3.1.2. WFE from userland</a></li>
<li><a href="#armv8-spinlock-pattern">33.10.3.1.3. ARMv8 spinlock pattern</a></li>
<li><a href="#gem5-arm-wfe">33.10.3.1.4. gem5 ARM WFE</a></li>
<li><a href="#arm-yield-instruction">33.10.3.1.5. ARM YIELD instruction</a></li>
</ul>
</li>
<li><a href="#arm-ldaxr-and-stlxr-instructions">33.10.3.2. ARM LDAXR and STLXR instructions</a></li>
<li><a href="#arm-psci">33.10.3.3. ARM PSCI</a></li>
<li><a href="#arm-dmb-instruction">33.10.3.4. ARM DMB instruction</a></li>
</ul>
</li>
<li><a href="#arm-timer">33.10.4. ARM timer</a></li>
<li><a href="#arm-gic">33.10.5. ARM GIC</a></li>
<li><a href="#arm-paging">33.10.6. ARM paging</a></li>
<li><a href="#arm-baremetal-bibliography">33.10.7. ARM baremetal bibliography</a>
<ul class="sectlevel4">
<li><a href="#nienfengyaoarmv8-bare-metal">33.10.7.1. NienfengYao/armv8-bare-metal</a></li>
<li><a href="#tukl-msdgem5-bare-metal">33.10.7.2. tukl-msd/gem5.bare-metal</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#how-we-got-some-baremetal-stuff-to-work">33.11. How we got some baremetal stuff to work</a>
<ul class="sectlevel3">
<li><a href="#find-the-uart-address">33.11.1. Find the UART address</a></li>
<li><a href="#aarch64-baremetal-neon-setup">33.11.2. aarch64 baremetal NEON setup</a></li>
</ul>
</li>
<li><a href="#baremetal-tests">33.12. Baremetal tests</a></li>
</ul>
</li>
<li><a href="#android">34. Android</a>
<ul class="sectlevel2">
<li><a href="#android-image-structure">34.1. Android image structure</a>
<ul class="sectlevel3">
<li><a href="#android-images-read-only">34.1.1. Android images read-only</a></li>
<li><a href="#android-data-partition">34.1.2. Android /data partition</a></li>
</ul>
</li>
<li><a href="#install-android-apps">34.2. Install Android apps</a></li>
<li><a href="#android-init">34.3. Android init</a></li>
</ul>
</li>
<li><a href="#benchmark-this-repo">35. Benchmark this repo</a>
<ul class="sectlevel2">
<li><a href="#continuous-integration">35.1. Continuous integration</a>
<ul class="sectlevel3">
<li><a href="#travis">35.1.1. Travis</a></li>
<li><a href="#circleci">35.1.2. CircleCI</a></li>
</ul>
</li>
<li><a href="#benchmark-this-repo-benchmarks">35.2. Benchmark this repo benchmarks</a>
<ul class="sectlevel3">
<li><a href="#benchmark-linux-kernel-boot">35.2.1. Benchmark Linux kernel boot</a>
<ul class="sectlevel4">
<li><a href="#gem5-arm-hpi-boot-takes-much-longer-than-aarch64">35.2.1.1. gem5 arm HPI boot takes much longer than aarch64</a></li>
<li><a href="#gem5-x86-64-derivo3cpu-boot-panics">35.2.1.2. gem5 x86_64 DerivO3CPU boot panics</a></li>
</ul>
</li>
<li><a href="#benchmark-emulators-on-userland-executables">35.2.2. Benchmark emulators on userland executables</a>
<ul class="sectlevel4">
<li><a href="#user-mode-vs-full-system-benchmark">35.2.2.1. User mode vs full system benchmark</a></li>
</ul>
</li>
<li><a href="#benchmark-builds">35.2.3. Benchmark builds</a>
<ul class="sectlevel4">
<li><a href="#find-which-buildroot-packages-are-making-the-build-slow-and-big">35.2.3.1. Find which Buildroot packages are making the build slow and big</a>
<ul class="sectlevel5">
<li><a href="#prebuilt-toolchain">35.2.3.1.1. Buildroot use prebuilt host toolchain</a></li>
</ul>
</li>
<li><a href="#benchmark-buildroot-build-baseline">35.2.3.2. Benchmark Buildroot build baseline</a></li>
<li><a href="#benchmark-gem5-build">35.2.3.3. Benchmark gem5 build</a>
<ul class="sectlevel5">
<li><a href="#pybind11-accounts-for-50-of-gem5-build-time">35.2.3.3.1. pybind11 accounts for 50% of gem5 build time</a></li>
<li><a href="#benchmark-gem5-single-file-change-rebuild-time">35.2.3.3.2. Benchmark gem5 single file change rebuild time</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#benchmark-machines">35.3. Benchmark machines</a>
<ul class="sectlevel3">
<li><a href="#p51">35.3.1. 2017 Lenovo ThinkPad P51</a>
<ul class="sectlevel4">
<li><a href="#p51-benchmarks">35.3.1.1. P51 benchmarks</a></li>
<li><a href="#intel-core-i7-7820hq-cpu">35.3.1.2. Intel Core i7-7820HQ CPU</a></li>
<li><a href="#samsung-m471a2k43bb1-crc-16gb-dram">35.3.1.3. Samsung M471A2K43BB1-CRC 16GB DRAM</a></li>
<li><a href="#samsung-mzvlb512hajq-000l7-512gb-ssd">35.3.1.4. Samsung MZVLB512HAJQ-000L7 512GB SSD</a></li>
<li><a href="#seagate-st1000lm035-1rk1-1tb-hard-disk">35.3.1.5. Seagate ST1000LM035-1RK1 1TB hard disk</a></li>
<li><a href="#nvidia-quadro-m1200-4gb-gddr5-gpu">35.3.1.6. NVIDIA Quadro M1200 4GB GDDR5 GPU</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#benchmark-internets">35.4. Benchmark Internets</a>
<ul class="sectlevel3">
<li><a href="#38mbps-internet">35.4.1. 38Mbps internet</a></li>
</ul>
</li>
<li><a href="#benchmark-this-repo-bibliography">35.5. Benchmark this repo bibliography</a></li>
</ul>
</li>
<li><a href="#compilers">36. Compilers</a>
<ul class="sectlevel2">
<li><a href="#prevent-statement-reordering">36.1. Prevent statement reordering</a></li>
<li><a href="#c-busy-loop">36.2. C busy loop</a></li>
</ul>
</li>
<li><a href="#computer-architecture">37. Computer architecture</a>
<ul class="sectlevel2">
<li><a href="#instruction-pipelining">37.1. Instruction pipelining</a>
<ul class="sectlevel3">
<li><a href="#classic-risc-pipeline">37.1.1. Classic RISC pipeline</a></li>
</ul>
</li>
<li><a href="#superscalar-processor">37.2. Superscalar processor</a>
<ul class="sectlevel3">
<li><a href="#execution-unit">37.2.1. Execution unit</a></li>
</ul>
</li>
<li><a href="#out-of-order-execution">37.3. Out-of-order execution</a>
<ul class="sectlevel3">
<li><a href="#speculative-execution">37.3.1. Speculative execution</a>
<ul class="sectlevel4">
<li><a href="#branch-predictor">37.3.1.1. Branch predictor</a></li>
</ul>
</li>
<li><a href="#re-order-buffer">37.3.2. Re-order buffer</a></li>
<li><a href="#register-renaming">37.3.3. Register renaming</a></li>
</ul>
</li>
<li><a href="#instruction-level-parallelism">37.4. Instruction level parallelism</a></li>
<li><a href="#hardware-threads">37.5. Hardware threads</a></li>
<li><a href="#caches">37.6. Caches</a>
<ul class="sectlevel3">
<li><a href="#cache-coherence">37.6.1. Cache coherence</a>
<ul class="sectlevel4">
<li><a href="#memory-consistency">37.6.1.1. Memory consistency</a>
<ul class="sectlevel5">
<li><a href="#sequential-consistency">37.6.1.1.1. Sequential Consistency</a></li>
</ul>
</li>
<li><a href="#can-caches-snoop-data-from-other-caches">37.6.1.2. Can caches snoop data from other caches?</a></li>
<li><a href="#vi-cache-coherence-protocol">37.6.1.3. VI cache coherence protocol</a></li>
<li><a href="#msi-cache-coherence-protocol">37.6.1.4. MSI cache coherence protocol</a>
<ul class="sectlevel5">
<li><a href="#msi-cache-coherence-protocol-with-transient-states">37.6.1.4.1. MSI cache coherence protocol with transient states</a></li>
</ul>
</li>
<li><a href="#mesi-cache-coherence-protocol">37.6.1.5. MESI cache coherence protocol</a></li>
<li><a href="#mosi-cache-coherence-protocol">37.6.1.6. MOSI cache coherence protocol</a></li>
<li><a href="#moesi">37.6.1.7. MOESI cache coherence protocol</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li><a href="#about-this-repo">38. About this repo</a>
<ul class="sectlevel2">
<li><a href="#supported-hosts">38.1. Supported hosts</a></li>
<li><a href="#common-build-issues">38.2. Common build issues</a>
<ul class="sectlevel3">
<li><a href="#put-source-uris-in-sources">38.2.1. You must put some 'source' URIs in your sources.list</a></li>
<li><a href="#build-from-downloaded-source-zip-files">38.2.2. Build from downloaded source zip files</a></li>
</ul>
</li>
<li><a href="#run-command-after-boot">38.3. Run command after boot</a></li>
<li><a href="#default-command-line-arguments">38.4. Default command line arguments</a></li>
<li><a href="#documentation">38.5. Documentation</a>
<ul class="sectlevel3">
<li><a href="#documentation-verification">38.5.1. Documentation verification</a>
<ul class="sectlevel4">
<li><a href="#asciidoctor-extract-link-targets">38.5.1.1. asciidoctor/extract-link-targets</a></li>
<li><a href="#asciidoctor-extract-header-ids">38.5.1.2. asciidoctor/extract-header-ids</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#asciidoctor-link-target-up-rb">38.6. asciidoctor/link-target-up.rb</a>
<ul class="sectlevel3">
<li><a href="#github-pages">38.6.1. GitHub pages</a></li>
</ul>
</li>
<li><a href="#clean-the-build">38.7. Clean the build</a></li>
<li><a href="#custom-build-directory">38.8. Custom build directory</a></li>
<li><a href="#ccache">38.9. ccache</a></li>
<li><a href="#getvar">38.10. getvar</a>
<ul class="sectlevel3">
<li><a href="#run-toolchain">38.10.1. run-toolchain</a>
<ul class="sectlevel4">
<li><a href="#disas">38.10.1.1. disas</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#rebuild-buildroot-while-running">38.11. Rebuild Buildroot while running</a></li>
<li><a href="#simultaneous-runs">38.12. Simultaneous runs</a></li>
<li><a href="#build-variants">38.13. Build variants</a>
<ul class="sectlevel3">
<li><a href="#linux-kernel-build-variants">38.13.1. Linux kernel build variants</a></li>
<li><a href="#qemu-build-variants">38.13.2. QEMU build variants</a></li>
<li><a href="#gem5-build-variants">38.13.3. gem5 build variants</a>
<ul class="sectlevel4">
<li><a href="#gem5-worktree">38.13.3.1. gem5 worktree</a></li>
<li><a href="#gem5-private-source-trees">38.13.3.2. gem5 private source trees</a></li>
</ul>
</li>
<li><a href="#buildroot-build-variants">38.13.4. Buildroot build variants</a></li>
</ul>
</li>
<li><a href="#optimization-level-of-a-build">38.14. Optimization level of a build</a></li>
<li><a href="#directory-structure">38.15. Directory structure</a>
<ul class="sectlevel3">
<li><a href="#lkmc-directory">38.15.1. lkmc directory</a>
<ul class="sectlevel4">
<li><a href="#userland-objects-vs-header-only">38.15.1.1. Userland objects vs header-only</a></li>
</ul>
</li>
<li><a href="#buildroot-packages-directory">38.15.2. buildroot_packages directory</a>
<ul class="sectlevel4">
<li><a href="#kernel-modules-buildroot-package">38.15.2.1. kernel_modules buildroot package</a></li>
</ul>
</li>
<li><a href="#patches-directory">38.15.3. patches directory</a>
<ul class="sectlevel4">
<li><a href="#patches-global-directory">38.15.3.1. patches/global directory</a></li>
<li><a href="#patches-manual-directory">38.15.3.2. patches/manual directory</a></li>
</ul>
</li>
<li><a href="#rootfs-overlay">38.15.4. rootfs_overlay</a>
<ul class="sectlevel4">
<li><a href="#out-rootfs-overlay-dir">38.15.4.1. <code>out_rootfs_overlay_dir</code></a>
<ul class="sectlevel5">
<li><a href="#disk-image-2">38.15.4.1.1. <code>disk_image_2</code></a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#lkmc-c">38.15.5. lkmc.c</a></li>
<li><a href="#lkmc-home">38.15.6. lkmc_home</a></li>
<li><a href="#path-properties">38.15.7. path_properties.py</a></li>
<li><a href="#rand-check-out">38.15.8. rand_check.out</a></li>
</ul>
</li>
<li><a href="#test-this-repo">38.16. Test this repo</a>
<ul class="sectlevel3">
<li><a href="#automated-tests">38.16.1. Automated tests</a>
<ul class="sectlevel4">
<li><a href="#test-arch-and-emulator-selection">38.16.1.1. Test arch and emulator selection</a></li>
<li><a href="#quit-on-fail">38.16.1.2. Quit on fail</a></li>
<li><a href="#test-userland-in-full-system">38.16.1.3. Test userland in full system</a></li>
<li><a href="#gdb-tests">38.16.1.4. GDB tests</a></li>
<li><a href="#magic-failure-string">38.16.1.5. Magic failure string</a></li>
</ul>
</li>
<li><a href="#non-automated-tests">38.16.2. Non-automated tests</a>
<ul class="sectlevel4">
<li><a href="#test-gdb-linux-kernel">38.16.2.1. Test GDB Linux kernel</a></li>
<li><a href="#test-the-internet">38.16.2.2. Test the Internet</a></li>
<li><a href="#cli-script-tests">38.16.2.3. CLI script tests</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#bisection">38.17. Bisection</a></li>
<li><a href="#update-a-forked-submodule">38.18. Update a forked submodule</a></li>
<li><a href="#release">38.19. Release</a>
<ul class="sectlevel3">
<li><a href="#release-procedure">38.19.1. Release procedure</a></li>
<li><a href="#release-zip">38.19.2. release-zip</a></li>
<li><a href="#release-upload">38.19.3. release-upload</a></li>
</ul>
</li>
<li><a href="#design-rationale">38.20. Design rationale</a>
<ul class="sectlevel3">
<li><a href="#design-goals">38.20.1. Design goals</a></li>
<li><a href="#setup-trade-offs">38.20.2. Setup trade-offs</a></li>
<li><a href="#resource-tradeoff-guidelines">38.20.3. Resource tradeoff guidelines</a></li>
<li><a href="#linux-distro-choice">38.20.4. Linux distro choice</a></li>
</ul>
</li>
<li><a href="#soft-topics">38.21. Soft topics</a>
<ul class="sectlevel3">
<li><a href="#fairy-tale">38.21.1. Fairy tale</a></li>
</ul>
</li>
<li><a href="#bibliography">38.22. Bibliography</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="china"><a class="anchor" href="#china"></a><a class="link" href="#china">1. <code>--china</code></a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The most important functionality of this repository is the <code>--china</code> option, sample usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./run --china &gt; index.html
firefox index.html</pre>
</div>
</div>
<div class="paragraph">
<p>see also: <a href="https://cirosantilli.com/china-dictatorship/mirrors" class="bare">https://cirosantilli.com/china-dictatorship/mirrors</a></p>
</div>
<div class="paragraph">
<p>The secondary systems programming functionality is described on the sections below starting from <a href="#getting-started">Getting started</a>.</p>
</div>
<div class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/china-dictatorship-media/master/Tiananmen_cute_girls.jpg" alt="Tiananmen cute girls" width="800">
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="getting-started"><a class="anchor" href="#getting-started"></a><a class="link" href="#getting-started">2. Getting started</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Each child section describes a possible different setup for this repo.</p>
</div>
<div class="paragraph">
<p>If you don&#8217;t know which one to go for, start with <a href="#qemu-buildroot-setup-getting-started">QEMU Buildroot setup getting started</a>.</p>
</div>
<div class="paragraph">
<p>Design goals of this project are documented at: <a href="#design-goals">Section 38.20.1, &#8220;Design goals&#8221;</a>.</p>
</div>
<div class="sect2">
<h3 id="should-you-waste-your-life-with-systems-programming"><a class="anchor" href="#should-you-waste-your-life-with-systems-programming"></a><a class="link" href="#should-you-waste-your-life-with-systems-programming">2.1. Should you waste your life with systems programming?</a></h3>
<div class="paragraph">
<p>Being the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn&#8217;t it?</p>
</div>
<div class="paragraph">
<p>But before you dedicate your life to this nonsense, do consider the following points:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>almost all contributions to the kernel are done by large companies, and if you are not an employee in one of them, you are likely not going to be able to do much.</p>
<div class="paragraph">
<p>This can be inferred by the fact that the <code>devices/</code> directory is by far the largest in the kernel.</p>
</div>
<div class="paragraph">
<p>The kernel is of course just an interface to hardware, and the hardware developers start developing their kernel stuff even before specs are publicly released, both to help with hardware development and to have things working when the announcement is made.</p>
</div>
<div class="paragraph">
<p>Furthermore, I believe that there are in-tree devices which have never been properly publicly documented. Linus is of course fine with this, since code == documentation for him, but it is not as easy for mere mortals.</p>
</div>
<div class="paragraph">
<p>There are some less hardware bound higher level layers in the kernel which might not require being in a hardware company, and a few people must be living off it.</p>
</div>
<div class="paragraph">
<p>But of course, those are heavily motivated by the underlying hardware characteristics, and it is very likely that most of the people working there were previously at a hardware company.</p>
</div>
<div class="paragraph">
<p>In that sense, therefore, the kernel is not as open as one might want to believe.</p>
</div>
<div class="paragraph">
<p>Of course, if there is some <a href="https://stackoverflow.com/questions/1697842/do-graphic-cards-have-instruction-sets-of-their-own/1697883">super useful and undocumented hardware that is just waiting there to be reverse engineered</a>, then that&#8217;s a much juicier target :-)</p>
</div>
</li>
<li>
<p>it is impossible to become rich with this knowledge.</p>
<div class="paragraph">
<p>This is partly implied by the fact that you need to be in a big company to make useful low level things, and therefore you will only be a tiny cog in the engine.</p>
</div>
<div class="paragraph">
<p>The key problem is that the entry cost of hardware design is just too insanely high for startups in general.</p>
</div>
</li>
<li>
<p>Is learning this the most useful thing that you think can do for society?</p>
<div class="paragraph">
<p>Or are you just learning it for job security and having a nice sounding title?</p>
</div>
<div class="paragraph">
<p>I&#8217;m not a huge fan of the person, but I think Jobs said it right: <a href="https://www.youtube.com/watch?v=FF-tKLISfPE" class="bare">https://www.youtube.com/watch?v=FF-tKLISfPE</a></p>
</div>
<div class="paragraph">
<p>First determine the useful goal, and then backtrack down to the most efficient thing you can do to reach it.</p>
</div>
</li>
<li>
<p>there are two things that sadden me compared to physics-based engineering:</p>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>you will never become eternally famous. All tech disappears sooner or later, while laws of nature, at least as useful approximations, stay unchanged.</p>
</li>
<li>
<p>every problem that you face is caused by imperfections introduced by other humans.</p>
<div class="paragraph">
<p>It is much easier to accept limitations of physics, and even natural selection in biology, which are not produced by a sentient being (?).</p>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>Physics-based engineering, just like low level hardware, is of course completely closed source however, since wrestling against the laws of physics is about the most expensive thing humans can do, so there&#8217;s also a downside to it.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Are you fine with those points, and ready to continue wasting your life with this crap?</p>
</div>
<div class="paragraph">
<p>Good. In that case, read on, and let&#8217;s have some fun together ;-)</p>
</div>
<div class="paragraph">
<p>Related: <a href="#soft-topics">Soft topics</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="qemu-buildroot-setup"><a class="anchor" href="#qemu-buildroot-setup"></a><a class="link" href="#qemu-buildroot-setup">2.2. QEMU Buildroot setup</a></h3>
<div class="sect3">
<h4 id="qemu-buildroot-setup-getting-started"><a class="anchor" href="#qemu-buildroot-setup-getting-started"></a><a class="link" href="#qemu-buildroot-setup-getting-started">2.2.1. QEMU Buildroot setup getting started</a></h4>
<div class="paragraph">
<p>This setup has been tested on Ubuntu 20.04.</p>
</div>
<div class="paragraph">
<p>The Buildroot build is already broken on Ubuntu 21.04 onwards: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/155" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/155</a>, just use the <a href="#docker">Docker host setup</a> setup in that case. We could fix it on Ubuntu 21.04, but it will break again inevitably later on.</p>
</div>
<div class="paragraph">
<p>For other host operating systems see: <a href="#supported-hosts">Section 38.1, &#8220;Supported hosts&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Reserve 12Gb of disk and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./setup
./build --download-dependencies qemu-buildroot
./run</pre>
</div>
</div>
<div class="paragraph">
<p>You don&#8217;t need to clone recursively even though we have <code>.git</code> submodules: <code>download-dependencies</code> fetches just the submodules that you need for this build to save time.</p>
</div>
<div class="paragraph">
<p>If something goes wrong, see: <a href="#common-build-issues">Section 38.2, &#8220;Common build issues&#8221;</a> and use our issue tracker: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues</a></p>
</div>
<div class="paragraph">
<p>The initial build will take a while (30 minutes to 2 hours) to clone and build, see <a href="#benchmark-builds">Benchmark builds</a> for more details.</p>
</div>
<div class="paragraph">
<p>If you don&#8217;t want to wait, you could also try the following faster but much more limited methods:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#prebuilt">Prebuilt setup</a></p>
</li>
<li>
<p><a href="#host">Host kernel module setup</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>but you will soon find that they are simply not enough if you anywhere near serious about systems programming.</p>
</div>
<div class="paragraph">
<p>After <code>./run</code>, QEMU opens up leaving you in the <a href="#lkmc-home"><code>/lkmc/</code> directory</a>, and you can start playing with the kernel modules inside the simulated system:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod hello.ko
insmod hello2.ko
rmmod hello
rmmod hello2</pre>
</div>
</div>
<div class="paragraph">
<p>This should print to the screen:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>hello init
hello2 init
hello cleanup
hello2 cleanup</pre>
</div>
</div>
<div class="paragraph">
<p>which are <code>printk</code> messages from <code>init</code> and <code>cleanup</code> methods of those modules.</p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/hello.c">kernel_modules/hello.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/hello2.c">kernel_modules/hello2.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Quit QEMU with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-A X</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="#quit-qemu-from-text-mode">Section 14.1.1, &#8220;Quit QEMU from text mode&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>All available modules can be found in the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules">kernel_modules</a> directory.</p>
</div>
<div class="paragraph">
<p>It is super easy to build for different <a href="#cpu-architecture">CPU architectures</a>, just use the <code>--arch</code> option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --arch aarch64 --download-dependencies qemu-buildroot
./run --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>To avoid typing <code>--arch aarch64</code> many times, you can set the default arch as explained at: <a href="#default-command-line-arguments">Section 38.4, &#8220;Default command line arguments&#8221;</a></p>
</div>
<div class="paragraph">
<p>I now urge you to read the following sections which contain widely applicable information:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#run-command-after-boot">Run command after boot</a></p>
</li>
<li>
<p><a href="#clean-the-build">Clean the build</a></p>
</li>
<li>
<p><a href="#build-the-documentation">Build the documentation</a></p>
</li>
<li>
<p>Linux kernel</p>
<div class="ulist">
<ul>
<li>
<p><a href="#printk">printk</a></p>
</li>
<li>
<p><a href="#kernel-command-line-parameters">Kernel command line parameters</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Once you use <a href="#gdb">GDB step debug</a> and <a href="#tmux">tmux</a>, your terminal will look a bit like this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    1.451857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/s1│loading @0xffffffffc0000000: ../kernel_modules-1.0//timer.ko
[    1.454310] ledtrig-cpu: registered to indicate activity on CPUs             │(gdb) b lkmc_timer_callback
[    1.455621] usbcore: registered new interface driver usbhid                  │Breakpoint 1 at 0xffffffffc0000000: file /home/ciro/bak/git/linux-kernel-module
[    1.455811] usbhid: USB HID core driver                                      │-cheat/out/x86_64/buildroot/build/kernel_modules-1.0/./timer.c, line 28.
[    1.462044] NET: Registered protocol family 10                               │(gdb) c
[    1.467911] Segment Routing with IPv6                                        │Continuing.
[    1.468407] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver              │
[    1.470859] NET: Registered protocol family 17                               │Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 &lt;mytimer&gt;)
[    1.472017] 9pnet: Installing 9P2000 support                                 │    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.475461] sched_clock: Marking stable (1473574872, 0)-&gt;(1554017593, -80442)│kernel_modules-1.0/./timer.c:28
[    1.479419] ALSA device list:                                                │28      {
[    1.479567]   No soundcards found.                                           │(gdb) c
[    1.619187] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100                 │Continuing.
[    1.622954] ata2.00: configured for MWDMA2                                   │
[    1.644048] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ P5│Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 &lt;mytimer&gt;)
[    1.741966] tsc: Refined TSC clocksource calibration: 2904.010 MHz           │    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.742796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x29dc0f4s│kernel_modules-1.0/./timer.c:28
[    1.743648] clocksource: Switched to clocksource tsc                         │28      {
[    2.072945] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8043│(gdb) bt
[    2.078641] EXT4-fs (vda): couldn't mount as ext3 due to feature incompatibis│#0  lkmc_timer_callback (data=0xffffffffc0002000 &lt;mytimer&gt;)
[    2.080350] EXT4-fs (vda): mounting ext2 file system using the ext4 subsystem│    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    2.088978] EXT4-fs (vda): mounted filesystem without journal. Opts: (null)  │kernel_modules-1.0/./timer.c:28
[    2.089872] VFS: Mounted root (ext2 filesystem) readonly on device 254:0.    │#1  0xffffffff810ab494 in call_timer_fn (timer=0xffffffffc0002000 &lt;mytimer&gt;,
[    2.097168] devtmpfs: mounted                                                │    fn=0xffffffffc0000000 &lt;lkmc_timer_callback&gt;) at kernel/time/timer.c:1326
[    2.126472] Freeing unused kernel memory: 1264K                              │#2  0xffffffff810ab71f in expire_timers (head=&lt;optimized out&gt;,
[    2.126706] Write protecting the kernel read-only data: 16384k               │    base=&lt;optimized out&gt;) at kernel/time/timer.c:1363
[    2.129388] Freeing unused kernel memory: 2024K                              │#3  __run_timers (base=&lt;optimized out&gt;) at kernel/time/timer.c:1666
[    2.139370] Freeing unused kernel memory: 1284K                              │#4  run_timer_softirq (h=&lt;optimized out&gt;) at kernel/time/timer.c:1692
[    2.246231] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck isd│#5  0xffffffff81a000cc in __do_softirq () at kernel/softirq.c:285
[    2.259574] EXT4-fs (vda): re-mounted. Opts: block_validity,barrier,user_xatr│#6  0xffffffff810577cc in invoke_softirq () at kernel/softirq.c:365
hello S98                                                                       │#7  irq_exit () at kernel/softirq.c:405
                                                                                │#8  0xffffffff818021ba in exiting_irq () at ./arch/x86/include/asm/apic.h:541
Apr 15 23:59:23 login[49]: root login on 'console'                              │#9  smp_apic_timer_interrupt (regs=&lt;optimized out&gt;)
hello /root/.profile                                                            │    at arch/x86/kernel/apic/apic.c:1052
# insmod /timer.ko                                                              │#10 0xffffffff8180190f in apic_timer_interrupt ()
[    6.791945] timer: loading out-of-tree module taints kernel.                 │    at arch/x86/entry/entry_64.S:857
# [    7.821621] 4294894248                                                     │#11 0xffffffff82003df8 in init_thread_union ()
[    8.851385] 4294894504                                                       │#12 0x0000000000000000 in ?? ()
                                                                                │(gdb)</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="how-to-hack-stuff"><a class="anchor" href="#how-to-hack-stuff"></a><a class="link" href="#how-to-hack-stuff">2.2.2. How to hack stuff</a></h4>
<div class="paragraph">
<p>Besides a seamless <a href="#qemu-buildroot-setup-getting-started">initial build</a>, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.</p>
</div>
<div class="sect4">
<h5 id="your-first-linux-kernel-hack"><a class="anchor" href="#your-first-linux-kernel-hack"></a><a class="link" href="#your-first-linux-kernel-hack">2.2.2.1. Your first Linux kernel hack</a></h5>
<div class="paragraph">
<p>Let&#8217;s hack up the <a href="#linux-kernel-entry-point">Linux kernel entry point</a>, which is an easy place to start.</p>
</div>
<div class="paragraph">
<p>Open the file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim submodules/linux/init/main.c</pre>
</div>
</div>
<div class="paragraph">
<p>and find the <code>start_kernel</code> function, then add there a:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pr_info("I'VE HACKED THE LINUX KERNEL!!!");</pre>
</div>
</div>
<div class="paragraph">
<p>Then rebuild the Linux kernel, quit QEMU and reboot the modified kernel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux
./run</pre>
</div>
</div>
<div class="paragraph">
<p>and, surely enough, your message has appeared at the beginning of the boot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    0.000000] I'VE HACKED THE LINUX KERNEL!!!</pre>
</div>
</div>
<div class="paragraph">
<p>So you are now officially a Linux kernel hacker, way to go!</p>
</div>
<div class="paragraph">
<p>We could have used just <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build">build</a> to rebuild the kernel as in the <a href="#qemu-buildroot-setup-getting-started">initial build</a> instead of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-linux">build-linux</a>, but building just the required individual components is preferred during development:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>saves a few seconds from parsing Make scripts and reading timestamps</p>
</li>
<li>
<p>makes it easier to understand what is being done in more detail</p>
</li>
<li>
<p>allows passing more specific options to customize the build</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build">build</a> script is just a lightweight wrapper that calls the smaller build scripts, and you can see what <code>./build</code> does with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --dry-run</pre>
</div>
</div>
<div class="paragraph">
<p>see also: <a href="#dry-run">Dry run to get commands for your project</a>.</p>
</div>
<div class="paragraph">
<p>When you reach difficulties, QEMU makes it possible to easily GDB step debug the Linux kernel source code, see: <a href="#gdb">Section 3, &#8220;GDB step debug&#8221;</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="your-first-kernel-module-hack"><a class="anchor" href="#your-first-kernel-module-hack"></a><a class="link" href="#your-first-kernel-module-hack">2.2.2.2. Your first kernel module hack</a></h5>
<div class="paragraph">
<p>Edit <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/hello.c">kernel_modules/hello.c</a> to contain:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pr_info("hello init hacked\n");</pre>
</div>
</div>
<div class="paragraph">
<p>and rebuild with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules</pre>
</div>
</div>
<div class="paragraph">
<p>Now there are two ways to test it out: the fast way, and the safe way.</p>
</div>
<div class="paragraph">
<p>The fast way is, without quitting or rebooting QEMU, just directly re-insert the module with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko</pre>
</div>
</div>
<div class="paragraph">
<p>and the new <code>pr_info</code> message should now show on the terminal at the end of the boot.</p>
</div>
<div class="paragraph">
<p>This works because we have a <a href="#9p">9P</a> mount there setup by default, which mounts the host directory that contains the build outputs on the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls "$(./getvar out_rootfs_overlay_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>The fast method is slightly risky because your previously insmodded buggy kernel module attempt might have corrupted the kernel memory, which could affect future runs.</p>
</div>
<div class="paragraph">
<p>Such failures are however unlikely, and you should be fine if you don&#8217;t see anything weird happening.</p>
</div>
<div class="paragraph">
<p>The safe way, is to fist <a href="#rebuild-buildroot-while-running">quit QEMU</a>, rebuild the modules, put them in the root filesystem, and then reboot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules
./build-buildroot
./run --eval-after 'insmod hello.ko'</pre>
</div>
</div>
<div class="paragraph">
<p><code>./build-buildroot</code> is required after <code>./build-modules</code> because it re-generates the root filesystem with the modules that we compiled at <code>./build-modules</code>.</p>
</div>
<div class="paragraph">
<p>You can see that <code>./build</code> does that as well, by running:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --dry-run</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="#dry-run">Dry run to get commands for your project</a>.</p>
</div>
<div class="paragraph">
<p><code>--eval-after</code> is optional: you could just type <code>insmod hello.ko</code> in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.</p>
</div>
<div class="paragraph">
<p>If the guest and host are the same arch, typically x86_64, you can speed up boot further with <a href="#kvm">KVM</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kvm</pre>
</div>
</div>
<div class="paragraph">
<p>All of this put together makes the safe procedure acceptably fast for regular development as well.</p>
</div>
<div class="paragraph">
<p>It is also easy to GDB step debug kernel modules with our setup, see: <a href="#gdb-step-debug-kernel-module">Section 3.4, &#8220;GDB step debug kernel module&#8221;</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="your-first-glibc-hack"><a class="anchor" href="#your-first-glibc-hack"></a><a class="link" href="#your-first-glibc-hack">2.2.2.3. Your first glibc hack</a></h5>
<div class="paragraph">
<p>We use <a href="#libc-choice">glibc as our default libc now</a>, and it is tracked as an unmodified submodule at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/submodules/glibc">submodules/glibc</a>, at the exact same version that Buildroot has it, which can be found at: <a href="https://github.com/buildroot/buildroot/blob/2018.05/package/glibc/glibc.mk#L13">package/glibc/glibc.mk</a>. Buildroot 2018.05 applies no patches.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s hack up the <code>puts</code> function:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot -- glibc-reconfigure</pre>
</div>
</div>
<div class="paragraph">
<p>with the patch:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff --git a/libio/ioputs.c b/libio/ioputs.c
index 706b20b492..23185948f3 100644
--- a/libio/ioputs.c
+++ b/libio/ioputs.c
@@ -38,8 +38,9 @@ _IO_puts (const char *str)
   if ((_IO_vtable_offset (_IO_stdout) != 0
        || _IO_fwide (_IO_stdout, -1) == -1)
       &amp;&amp; _IO_sputn (_IO_stdout, str, len) == len
+      &amp;&amp; _IO_sputn (_IO_stdout, " hacked", 7) == 7
       &amp;&amp; _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
-    result = MIN (INT_MAX, len + 1);
+    result = MIN (INT_MAX, len + 1 + 7);

   _IO_release_lock (_IO_stdout);
   return result;</pre>
</div>
</div>
<div class="paragraph">
<p>And then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './c/hello.out'</pre>
</div>
</div>
<div class="paragraph">
<p>outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>hello hacked</pre>
</div>
</div>
<div class="paragraph">
<p>Lol!</p>
</div>
<div class="paragraph">
<p>We can also test our hacked glibc on <a href="#user-mode-simulation">User mode simulation</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>I just noticed that this is actually a good way to develop glibc for other archs.</p>
</div>
<div class="paragraph">
<p>In this example, we got away without recompiling the userland program because we made a change that did not affect the glibc ABI, see this answer for an introduction to ABI stability: <a href="https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743" class="bare">https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743</a></p>
</div>
<div class="paragraph">
<p>Note that for arch agnostic features that don&#8217;t rely on bleeding kernel changes that you host doesn&#8217;t yet have, you can develop glibc natively as explained at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/10412684/how-to-compile-my-own-glibc-c-standard-library-from-source-and-use-it/52454710#52454710" class="bare">https://stackoverflow.com/questions/10412684/how-to-compile-my-own-glibc-c-standard-library-from-source-and-use-it/52454710#52454710</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host/52454603#52454603" class="bare">https://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host/52454603#52454603</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version/52550158#52550158" class="bare">https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version/52550158#52550158</a> more focus on symbol versioning, but no one knows how to do it, so I answered</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested on a30ed0f047523ff2368d421ee2cce0800682c44e + 1.</p>
</div>
</div>
<div class="sect4">
<h5 id="your-first-binutils-hack"><a class="anchor" href="#your-first-binutils-hack"></a><a class="link" href="#your-first-binutils-hack">2.2.2.4. Your first Binutils hack</a></h5>
<div class="paragraph">
<p>Have you ever felt that a single <code>inc</code> instruction was not enough? Really? Me too!</p>
</div>
<div class="paragraph">
<p>So let&#8217;s hack the <a href="#gnu-gas-assembler">GNU GAS assembler</a>, which is part of <a href="https://en.wikipedia.org/wiki/GNU_Binutils">GNU Binutils</a>, to add a new shiny version of <code>inc</code> called&#8230;&#8203; <code>myinc</code>!</p>
</div>
<div class="paragraph">
<p>GCC uses GNU GAS as its backend, so we will test out new mnemonic with an <a href="#gcc-inline-assembly">GCC inline assembly</a> test program: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/binutils_hack.c">userland/arch/x86_64/binutils_hack.c</a>, which is just a copy of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/binutils_nohack.c">userland/arch/x86_64/binutils_nohack.c</a> but with <code>myinc</code> instead of <code>inc</code>.</p>
</div>
<div class="paragraph">
<p>The inline assembly is disabled with an <code>#ifdef</code>, so first modify the source to enable that.</p>
</div>
<div class="paragraph">
<p>Then, try to build userland:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland</pre>
</div>
</div>
<div class="paragraph">
<p>and watch it fail with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>binutils_hack.c:8: Error: no such instruction: `myinc %rax'</pre>
</div>
</div>
<div class="paragraph">
<p>Now, edit the file</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim submodules/binutils-gdb/opcodes/i386-tbl.h</pre>
</div>
</div>
<div class="paragraph">
<p>and add a copy of the <code>"inc"</code> instruction just next to it, but with the new name <code>"myinc"</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h
index af583ce578..3cc341f303 100644
--- a/opcodes/i386-tbl.h
+++ b/opcodes/i386-tbl.h
@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =
     { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
 	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
+  { "myinc", 1, 0xfe, 0x0, 1,
+    { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },
+    { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0 },
+    { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
+	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
   { "sub", 2, 0x28, None, 1,
     { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,</pre>
</div>
</div>
<div class="paragraph">
<p>Finally, rebuild Binutils, userland and test our program with <a href="#user-mode-simulation">User mode simulation</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot -- host-binutils-rebuild
./build-userland --static
./run --static --userland userland/arch/x86_64/binutils_hack.c</pre>
</div>
</div>
<div class="paragraph">
<p>and we se that <code>myinc</code> worked since the assert did not fail!</p>
</div>
<div class="paragraph">
<p>Tested on b60784d59bee993bf0de5cde6c6380dd69420dda + 1.</p>
</div>
</div>
<div class="sect4">
<h5 id="your-first-gcc-hack"><a class="anchor" href="#your-first-gcc-hack"></a><a class="link" href="#your-first-gcc-hack">2.2.2.5. Your first GCC hack</a></h5>
<div class="paragraph">
<p>OK, now time to hack GCC.</p>
</div>
<div class="paragraph">
<p>For convenience, let&#8217;s use the <a href="#user-mode-simulation">User mode simulation</a>.</p>
</div>
<div class="paragraph">
<p>If we run the program <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/gcc_hack.c">userland/c/gcc_hack.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --static
./run --static --userland userland/c/gcc_hack.c</pre>
</div>
</div>
<div class="paragraph">
<p>it produces the normal boring output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i = 2
j = 0</pre>
</div>
</div>
<div class="paragraph">
<p>So how about we swap <code>++</code> and <code>--</code> to make things more fun?</p>
</div>
<div class="paragraph">
<p>Open the file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim submodules/gcc/gcc/c/c-parser.c</pre>
</div>
</div>
<div class="paragraph">
<p>and find the function <code>c_parser_postfix_expression_after_primary</code>.</p>
</div>
<div class="paragraph">
<p>In that function, swap <code>case CPP_PLUS_PLUS</code> and <code>case CPP_MINUS_MINUS</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff --git a/gcc/c/c-parser.c b/gcc/c/c-parser.c
index 101afb8e35f..89535d1759a 100644
--- a/gcc/c/c-parser.c
+++ b/gcc/c/c-parser.c
@@ -8529,7 +8529,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 		expr.original_type = DECL_BIT_FIELD_TYPE (field);
 	    }
 	  break;
-	case CPP_PLUS_PLUS:
+	case CPP_MINUS_MINUS:
 	  /* Postincrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)-&gt;get_finish ();
@@ -8548,7 +8548,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 	  expr.original_code = ERROR_MARK;
 	  expr.original_type = NULL;
 	  break;
-	case CPP_MINUS_MINUS:
+	case CPP_PLUS_PLUS:
 	  /* Postdecrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)-&gt;get_finish ();</pre>
</div>
</div>
<div class="paragraph">
<p>Now rebuild GCC, the program and re-run it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot -- host-gcc-final-rebuild
./build-userland --static
./run --static --userland userland/c/gcc_hack.c</pre>
</div>
</div>
<div class="paragraph">
<p>and the new ouptut is now:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i = 2
j = 0</pre>
</div>
</div>
<div class="paragraph">
<p>We need to use the ugly <code>-final</code> thing because GCC has to packages in Buildroot, <code>-initial</code> and <code>-final</code>: <a href="https://stackoverflow.com/questions/54992977/how-to-select-an-override-srcdir-source-for-gcc-when-building-buildroot" class="bare">https://stackoverflow.com/questions/54992977/how-to-select-an-override-srcdir-source-for-gcc-when-building-buildroot</a> No one is able to example precisely with a minimal example why this is required:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/39883865/why-multiple-passes-for-building-linux-from-scratch-lfs" class="bare">https://stackoverflow.com/questions/39883865/why-multiple-passes-for-building-linux-from-scratch-lfs</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/27457835/why-do-cross-compilers-have-a-two-stage-compilation" class="bare">https://stackoverflow.com/questions/27457835/why-do-cross-compilers-have-a-two-stage-compilation</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="about-the-qemu-buildroot-setup"><a class="anchor" href="#about-the-qemu-buildroot-setup"></a><a class="link" href="#about-the-qemu-buildroot-setup">2.2.3. About the QEMU Buildroot setup</a></h4>
<div class="paragraph">
<p>What QEMU and Buildroot are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#introduction-to-buildroot">Introduction to Buildroot</a></p>
</li>
<li>
<p><a href="#introduction-to-qemu">Introduction to QEMU</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is our reference setup, and the best supported one, use it unless you have good reason not to.</p>
</div>
<div class="paragraph">
<p>It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.</p>
</div>
<div class="paragraph">
<p>Read the following sections for further introductory material:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#introduction-to-qemu">Introduction to QEMU</a></p>
</li>
<li>
<p><a href="#introduction-to-buildroot">Introduction to Buildroot</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="dry-run"><a class="anchor" href="#dry-run"></a><a class="link" href="#dry-run">2.3. Dry run to get commands for your project</a></h3>
<div class="paragraph">
<p>One of the major features of this repository is that we try to support the <code>--dry-run</code> option really well for all scripts.</p>
</div>
<div class="paragraph">
<p>This option, as the name suggests, outputs the external commands that would be run (or more precisely: equivalent commands), without actually running them.</p>
</div>
<div class="paragraph">
<p>This allows you to just clone this repository and get full working commands to integrate into your project, without having to build or use this setup further!</p>
</div>
<div class="paragraph">
<p>For example, we can obtain a QEMU run for the file <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a> in <a href="#user-mode-simulation">User mode simulation</a> by adding <code>--dry-run</code> to the normal command:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --dry-run --userland userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>which as of LKMC a18f28e263c91362519ef550150b5c9d75fa3679 + 1 outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>+ /path/to/linux-kernel-module-cheat/out/qemu/default/opt/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  -r 5.2.1 \
  -seed 0 \
  -trace enable=load_file,file=/path/to/linux-kernel-module-cheat/out/run/qemu/x86_64/0/trace.bin \
  -cpu max \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out \
;</pre>
</div>
</div>
<div class="paragraph">
<p>So observe that the command contains:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>+</code>: sign to differentiate it from program stdout, much like bash <code>-x</code> output. This is not a valid part of the generated Bash command however.</p>
</li>
<li>
<p>the actual command nicely, indented and with arguments broken one per line, but with continuing backslashes so you can just copy paste into a terminal</p>
<div class="paragraph">
<p>For setups that don&#8217;t support the newline e.g. <a href="#gem5-eclipse-configuration">Eclipse debugging</a>, you can turn them off with <code>--print-cmd-oneline</code></p>
</div>
</li>
<li>
<p><code>;</code>: both a valid part of the Bash command, and a visual mark the end of the command</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For the specific case of running emulators such as QEMU, the last command is also automatically placed in a file for your convenience and later inspection:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar run_dir)/run.sh"</pre>
</div>
</div>
<div class="paragraph">
<p>Since we need this so often, the last run command is also stored for convenience at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat out/run.sh</pre>
</div>
</div>
<div class="paragraph">
<p>although this won&#8217;t of course work well for <a href="#simultaneous-runs">Simultaneous runs</a>.</p>
</div>
<div class="paragraph">
<p>Furthermore, <code>--dry-run</code> also automatically specifies, in valid Bash shell syntax:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>environment variables used to run the command with syntax <code>+ ENV_VAR_1=abc ENV_VAR_2=def ./some/command</code></p>
</li>
<li>
<p>change in working directory with <code>+ cd /some/new/path &amp;&amp; ./some/command</code></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="gem5-buildroot-setup"><a class="anchor" href="#gem5-buildroot-setup"></a><a class="link" href="#gem5-buildroot-setup">2.4. gem5 Buildroot setup</a></h3>
<div class="sect3">
<h4 id="about-the-gem5-buildroot-setup"><a class="anchor" href="#about-the-gem5-buildroot-setup"></a><a class="link" href="#about-the-gem5-buildroot-setup">2.4.1. About the gem5 Buildroot setup</a></h4>
<div class="paragraph">
<p>This setup is like the <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a>, but it uses <a href="http://gem5.org/">gem5</a> instead of QEMU as a system simulator.</p>
</div>
<div class="paragraph">
<p>QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.</p>
</div>
<div class="paragraph">
<p>The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.</p>
</div>
<div class="paragraph">
<p>gem5 on the other hand, can simulate the system in more detail than QEMU, including:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>simplified CPU pipeline</p>
</li>
<li>
<p>caches</p>
</li>
<li>
<p>DRAM timing</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and can therefore be used to estimate system performance, see: <a href="#gem5-run-benchmark">Section 24.2, &#8220;gem5 run benchmark&#8221;</a> for an example.</p>
</div>
<div class="paragraph">
<p>The downside of gem5 much slower than QEMU because of the greater simulation detail.</p>
</div>
<div class="paragraph">
<p>See <a href="#gem5-vs-qemu">gem5 vs QEMU</a> for a more thorough comparison.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-buildroot-setup-getting-started"><a class="anchor" href="#gem5-buildroot-setup-getting-started"></a><a class="link" href="#gem5-buildroot-setup-getting-started">2.4.2. gem5 Buildroot setup getting started</a></h4>
<div class="paragraph">
<p>For the most part, if you just add the <code>--emulator gem5</code> option or <code>*-gem5</code> suffix to all commands and everything should magically work.</p>
</div>
<div class="paragraph">
<p>If you haven&#8217;t built Buildroot yet for <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a>, you can build from the beginning with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --download-dependencies gem5-buildroot
./run --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>If you have already built previously, don&#8217;t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so <code>./build</code> will be fast.</p>
</div>
<div class="paragraph">
<p>Remember that the gem5 boot is <a href="#benchmark-linux-kernel-boot">considerably slower</a> than QEMU since the simulation is more detailed.</p>
</div>
<div class="paragraph">
<p>If you have a relatively new GCC version and the gem5 build fails on your machine, see: <a href="#gem5-build-broken-on-recent-compiler-version">gem5 build broken on recent compiler version</a>.</p>
</div>
<div class="paragraph">
<p>To get a terminal, either open a new shell and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-shell</pre>
</div>
</div>
<div class="paragraph">
<p>You can quit the shell without killing gem5 by typing tilde followed by a period:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>~.</pre>
</div>
</div>
<div class="paragraph">
<p>If you are inside <a href="#tmux">tmux</a>, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --tmux</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="#tmux-gem5">Section 3.3.1, &#8220;tmux gem5&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>At the end of boot, it might not be very clear that you have the shell since some <a href="#printk">printk</a> messages may appear in front of the prompt like this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># &lt;6&gt;[    1.215329] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1cd486fa865, max_idle_ns: 440795259574 ns
&lt;6&gt;[    1.215351] clocksource: Switched to clocksource tsc</pre>
</div>
</div>
<div class="paragraph">
<p>but if you look closely, the <code>PS1</code> prompt marker <code>#</code> is there already, just hit enter and a clear prompt line will appear.</p>
</div>
<div class="paragraph">
<p>If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar --emulator gem5 m5out_dir)/system.pc.com_1.device"</pre>
</div>
</div>
<div class="paragraph">
<p>More gem5 information is present at: <a href="#gem5">Section 24, &#8220;gem5&#8221;</a></p>
</div>
<div class="paragraph">
<p>Good next steps are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-run-benchmark">gem5 run benchmark</a>: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU</p>
</li>
<li>
<p><a href="#m5out-directory">m5out directory</a>: understand the output files that gem5 produces, which contain information about your run</p>
</li>
<li>
<p><a href="#m5ops">m5ops</a>: magic guest instructions used to control gem5</p>
</li>
<li>
<p><a href="#add-new-files-to-the-buildroot-image">Add new files to the Buildroot image</a>: how to add your own files to the image if you have a benchmark that we don&#8217;t already support out of the box (also send a pull request!)</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="docker"><a class="anchor" href="#docker"></a><a class="link" href="#docker">2.5. Docker host setup</a></h3>
<div class="paragraph">
<p>This repository has been tested inside clean <a href="https://en.wikipedia.org/wiki/Docker_(software)">Docker</a> containers.</p>
</div>
<div class="paragraph">
<p>This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: <a href="#supported-hosts">Section 38.1, &#8220;Supported hosts&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>For example, to do a <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a> inside Docker, run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install docker
./run-docker create &amp;&amp; \
./run-docker sh -- ./build --download-dependencies qemu-buildroot
./run-docker</pre>
</div>
</div>
<div class="paragraph">
<p>You are now left inside a shell in the Docker! From there, just run as usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>The host git top level directory is mounted inside the guest with a <a href="https://stackoverflow.com/questions/23439126/how-to-mount-a-host-directory-in-a-docker-container">Docker volume</a>, which means for example that you can use your host&#8217;s GUI text editor directly on the files. Just don&#8217;t forget that if you nuke that directory on the guest, then it gets nuked on the host as well!</p>
</div>
<div class="paragraph">
<p>Command breakdown:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>./run-docker create</code>: create the image and container.</p>
<div class="paragraph">
<p>Needed only the very first time you use Docker, or if you run <code>./run-docker DESTROY</code> to restart for scratch, or save some disk space.</p>
</div>
<div class="paragraph">
<p>The image and container name is <code>lkmc</code>. The container shows under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>docker ps -a</pre>
</div>
</div>
<div class="paragraph">
<p>and the image shows under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>docker images</pre>
</div>
</div>
</li>
<li>
<p><code>./run-docker</code>: open a shell on the container.</p>
<div class="paragraph">
<p>If it has not been started previously, start it. This can also be done explicitly with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-docker start</pre>
</div>
</div>
<div class="paragraph">
<p>Quit the shell as usual with <code>Ctrl-D</code></p>
</div>
<div class="paragraph">
<p>This can be called multiple times from different host terminals to open multiple shells.</p>
</div>
</li>
<li>
<p><code>./run-docker stop</code>: stop the container.</p>
<div class="paragraph">
<p>This might save a bit of CPU and RAM once you stop working on this project, but it should not be a lot.</p>
</div>
</li>
<li>
<p><code>./run-docker DESTROY</code>: delete the container and image.</p>
<div class="paragraph">
<p>This doesn&#8217;t really clean the build, since we mount the guest&#8217;s working directory on the host git top-level, so you basically just got rid of the <code>apt-get</code> installs.</p>
</div>
<div class="paragraph">
<p>To actually delete the Docker build, run on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># sudo rm -rf out.docker</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>To use <a href="#gdb">GDB step debug</a> from inside Docker, you need a second shell inside the container. You can either do that from another shell with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-docker</pre>
</div>
</div>
<div class="paragraph">
<p>or even better, by starting a <a href="#tmux">tmux</a> session inside the container. We install <code>tmux</code> by default in the container.</p>
</div>
<div class="paragraph">
<p>You can also start a second shell and run a command in it at the same time with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-docker sh -- ./run-gdb start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>To use <a href="#qemu-graphic-mode">QEMU graphic mode</a> from Docker, run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --graphic --vnc</pre>
</div>
</div>
<div class="paragraph">
<p>and then on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install vinagre
./vnc</pre>
</div>
</div>
<div class="paragraph">
<p>TODO make files created inside Docker be owned by the current user in host instead of <code>root</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/33681396/how-do-i-write-to-a-volume-container-as-non-root-in-docker" class="bare">https://stackoverflow.com/questions/33681396/how-do-i-write-to-a-volume-container-as-non-root-in-docker</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes" class="bare">https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/31779802/shared-volume-file-permissions-ownership-docker" class="bare">https://stackoverflow.com/questions/31779802/shared-volume-file-permissions-ownership-docker</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="prebuilt"><a class="anchor" href="#prebuilt"></a><a class="link" href="#prebuilt">2.6. Prebuilt setup</a></h3>
<div class="sect3">
<h4 id="about-the-prebuilt-setup"><a class="anchor" href="#about-the-prebuilt-setup"></a><a class="link" href="#about-the-prebuilt-setup">2.6.1. About the prebuilt setup</a></h4>
<div class="paragraph">
<p>This setup uses prebuilt binaries that we upload to GitHub from time to time.</p>
</div>
<div class="paragraph">
<p>We don&#8217;t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.</p>
</div>
<div class="paragraph">
<p>Our prebuilts currently include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#qemu-buildroot-setup">QEMU Buildroot setup</a> binaries</p>
<div class="ulist">
<ul>
<li>
<p>Linux kernel</p>
</li>
<li>
<p>root filesystem</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="#baremetal-setup">Baremetal setup</a> binaries for QEMU</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For more details, see our our <a href="#release">release procedure</a>.</p>
</div>
<div class="paragraph">
<p>Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.</p>
</div>
<div class="paragraph">
<p>The limitations are severe however:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>can&#8217;t <a href="#gdb">GDB step debug the kernel</a>, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <a href="#prebuilt-toolchain">Section 35.2.3.1.1, &#8220;Buildroot use prebuilt host toolchain&#8221;</a>.</p>
<div class="paragraph">
<p>Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.</p>
</div>
</li>
<li>
<p>you won&#8217;t get the latest version of this repository. Our <a href="#travis">Travis</a> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyway.</p>
</li>
<li>
<p><a href="#gem5">gem5</a> is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses <code>vmlinux</code>, and once for QEMU which uses <code>arch/*</code> images, see also:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79</a></p>
</li>
<li>
<p><a href="#vmlinux-vs-bzimage-vs-zimage-vs-image">vmlinux vs bzImage vs zImage vs Image</a>.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the <a href="#qemu-buildroot-setup">full featured setup</a> while you take a coffee or a nap? :-)</p>
</div>
</div>
<div class="sect3">
<h4 id="prebuilt-setup-getting-started"><a class="anchor" href="#prebuilt-setup-getting-started"></a><a class="link" href="#prebuilt-setup-getting-started">2.6.2. Prebuilt setup getting started</a></h4>
<div class="paragraph">
<p>Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install qemu-system-x86
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
git checkout "$(git rev-list --tags --max-count=1)"
./release-download-latest
unzip lkmc-*.zip
./run --qemu-which host</pre>
</div>
</div>
<div class="paragraph">
<p>You have to checkout to the latest tag to ensure that the scripts match the release format: <a href="https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git" class="bare">https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git</a></p>
</div>
<div class="paragraph">
<p>This is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven&#8217;t investigated why.</p>
</div>
<div class="paragraph">
<p>Or to run a baremetal example instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --baremetal userland/c/hello.c \
  --qemu-which host \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Be saner and use our custom built QEMU instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --download-dependencies qemu
./run</pre>
</div>
</div>
<div class="paragraph">
<p>To build the kernel modules as in <a href="#your-first-kernel-module-hack">Your first kernel module hack</a> do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux --no-modules-install -- modules_prepare
./build-modules --gcc-which host
./run</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: for now the only way to test those modules out without <a href="#qemu-buildroot-setup-getting-started">building Buildroot</a> is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.</p>
</div>
<div class="paragraph">
<p>Command explanation:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>modules_prepare</code> does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.</p>
</li>
<li>
<p><code>--gcc-which host</code> selects your host Ubuntu packaged GCC, since you don&#8217;t have the Buildroot toolchain</p>
</li>
<li>
<p><code>--no-modules-install</code> is required otherwise the <code>make modules_install</code> target we run by default fails, since the kernel wasn&#8217;t built</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To modify the Linux kernel, build and use it as usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux
./run</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="host"><a class="anchor" href="#host"></a><a class="link" href="#host">2.7. Host kernel module setup</a></h3>
<div class="paragraph">
<p><strong>THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED</strong></p>
</div>
<div class="paragraph">
<p>This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.</p>
</div>
<div class="paragraph">
<p>It has however severe limitations:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>can&#8217;t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since <a href="https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681">the Linux kernel does not have a stable kernel module API</a>.</p>
</li>
<li>
<p>bugs can easily break you system. E.g.:</p>
<div class="ulist">
<ul>
<li>
<p>segfaults can trivially lead to a kernel crash, and require a reboot</p>
</li>
<li>
<p>your disk could get erased. Yes, this can also happen with <code>sudo</code> from userland. But you should not use <code>sudo</code> when developing newbie programs. And for the kernel you don&#8217;t have the choice not to use <code>sudo</code>.</p>
</li>
<li>
<p>even more subtle system corruption such as <a href="https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r">not being able to rmmod</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>can&#8217;t control which hardware is used, notably the CPU architecture</p>
</li>
<li>
<p>can&#8217;t step debug it with <a href="#gdb">GDB</a> easily. The alternatives are <a href="https://en.wikipedia.org/wiki/JTAG">JTAG</a> or <a href="#kgdb">KGDB</a>, but those are less reliable, and require extra hardware.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Still interested?</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules --host</pre>
</div>
</div>
<div class="paragraph">
<p>Compilation will likely fail for some modules because of kernel or toolchain differences that we can&#8217;t control on the host.</p>
</div>
<div class="paragraph">
<p>The best workaround is to compile just your modules with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules --host -- hello hello2</pre>
</div>
</div>
<div class="paragraph">
<p>which is equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules \
  --gcc-which host \
  --host \
  -- \
  kernel_modules/hello.c \
  kernel_modules/hello2.c \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Or just remove the <code>.c</code> extension from the failing files and try again:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd "$(./getvar kernel_modules_source_dir)"
mv broken.c broken.c~</pre>
</div>
</div>
<div class="paragraph">
<p>Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd "$(./getvar kernel_modules_build_host_subdir)"
sudo insmod hello.ko

# Our module is there.
sudo lsmod | grep hello

# Last message should be: hello init
dmesg -T

sudo rmmod hello

# Last message should be: hello exit
dmesg -T

# Not present anymore
sudo lsmod | grep hello</pre>
</div>
</div>
<div class="sect3">
<h4 id="hello-host"><a class="anchor" href="#hello-host"></a><a class="link" href="#hello-host">2.7.1. Hello host</a></h4>
<div class="paragraph">
<p>Minimal host build system example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd hello_host_kernel_module
make
sudo insmod hello.ko
dmesg
sudo rmmod hello.ko
dmesg</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="userland-setup"><a class="anchor" href="#userland-setup"></a><a class="link" href="#userland-setup">2.8. Userland setup</a></h3>
<div class="sect3">
<h4 id="about-the-userland-setup"><a class="anchor" href="#about-the-userland-setup"></a><a class="link" href="#about-the-userland-setup">2.8.1. About the userland setup</a></h4>
<div class="paragraph">
<p>In order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/">userland/</a></p>
</li>
<li>
<p><a href="#rootfs-overlay">rootfs_overlay</a></p>
</li>
<li>
<p><a href="#add-new-buildroot-packages">Add new Buildroot packages</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>When we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.</p>
</div>
<div class="paragraph">
<p>However, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.</p>
</div>
<div class="paragraph">
<p>Therefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.</p>
</div>
<div class="paragraph">
<p>Notable userland content included / moving into this repository includes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#userland-assembly">Userland assembly</a></p>
</li>
<li>
<p><a href="#c">C</a></p>
</li>
<li>
<p><a href="#cpp">C++</a></p>
</li>
<li>
<p><a href="#posix">POSIX</a></p>
</li>
<li>
<p><a href="#algorithms">Algorithms</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="userland-setup-getting-started"><a class="anchor" href="#userland-setup-getting-started"></a><a class="link" href="#userland-setup-getting-started">2.8.2. Userland setup getting started</a></h4>
<div class="paragraph">
<p>There are several ways to run our <a href="#userland-content">Userland content</a>, notably:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>natively on the host as shown at: <a href="#userland-setup-getting-started-natively">Section 2.8.2.1, &#8220;Userland setup getting started natively&#8221;</a></p>
<div class="paragraph">
<p>Can only run examples compatible with your host CPU architecture and OS, but has the fastest setup and runtimes.</p>
</div>
</li>
<li>
<p>from user mode simulation with:</p>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>the host prebuilt toolchain: <a href="#userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode">Section 2.8.2.2, &#8220;Userland setup getting started with prebuilt toolchain and QEMU user mode&#8221;</a></p>
</li>
<li>
<p>the Buildroot toolchain you built yourself: <a href="#qemu-user-mode-getting-started">Section 11.1, &#8220;QEMU user mode getting started&#8221;</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>This setup:</p>
</div>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>can run most examples, including those for other CPU architectures, with the notable exception of examples that rely on kernel modules</p>
</li>
<li>
<p>can run reproducible approximate performance experiments with gem5, see e.g. <a href="#bst-vs-heap-vs-hashmap">BST vs heap vs hashmap</a></p>
</li>
</ul>
</div>
</div>
</div>
</li>
<li>
<p>from full system simulation as shown at: <a href="#qemu-buildroot-setup-getting-started">Section 2.2.1, &#8220;QEMU Buildroot setup getting started&#8221;</a>.</p>
<div class="paragraph">
<p>This is the most reproducible and controlled environment, and all examples work there. But also the slower one to setup.</p>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="userland-setup-getting-started-natively"><a class="anchor" href="#userland-setup-getting-started-natively"></a><a class="link" href="#userland-setup-getting-started-natively">2.8.2.1. Userland setup getting started natively</a></h5>
<div class="paragraph">
<p>With this setup, we will use the host toolchain and execute executables directly on the host.</p>
</div>
<div class="paragraph">
<p>No toolchain build is required, so you can just download your distro toolchain and jump straight into it.</p>
</div>
<div class="paragraph">
<p>Build, run and example, and clean it in-tree with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install gcc
cd userland
./build c/hello
./c/hello.out
./build --clean</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a>.</p>
</div>
<div class="paragraph">
<p>Build an entire directory and test it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd userland
./build c
./test c</pre>
</div>
</div>
<div class="paragraph">
<p>Build the current directory and test it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd userland/c
./build
./test</pre>
</div>
</div>
<div class="paragraph">
<p>As mentioned at <a href="#userland-libs-directory">userland/libs directory</a>, tests under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs">userland/libs</a> require certain optional libraries to be installed, and are not built or tested by default.</p>
</div>
<div class="paragraph">
<p>You can install those libraries with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd linux-kernel-module-cheat
./setup
./build --download-dependencies userland-host</pre>
</div>
</div>
<div class="paragraph">
<p>and then build the examples and test with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --package-all
./test --package-all</pre>
</div>
</div>
<div class="paragraph">
<p>Pass custom compiler options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --ccflags='-foptimize-sibling-calls -foptimize-strlen' --force-rebuild</pre>
</div>
</div>
<div class="paragraph">
<p>Here we used <code>--force-rebuild</code> to force rebuild since the sources weren&#8217;t modified since the last build.</p>
</div>
<div class="paragraph">
<p>Some CLI options have more specialized flags, e.g. <code>-O</code> for the <a href="#optimization-level-of-a-build">Optimization level of a build</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --optimization-level 3 --force-rebuild</pre>
</div>
</div>
<div class="paragraph">
<p>See also <a href="#user-mode-static-executables">User mode static executables</a> for <code>--static</code>.</p>
</div>
<div class="paragraph">
<p>The <code>build</code> scripts inside <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/">userland/</a> are just symlinks to <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-userland-in-tree">build-userland-in-tree</a> which you can also use from toplevel as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland-in-tree
./build-userland-in-tree userland/c
./build-userland-in-tree userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p><code>build-userland-in-tree</code> is in turn just a thin wrapper around <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-userland">build-userland</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --gcc-which host --in-tree userland/c</pre>
</div>
</div>
<div class="paragraph">
<p>So you can use any option supported by <code>build-userland</code> script freely with <code>build-userland-in-tree</code> and <code>build</code>.</p>
</div>
<div class="paragraph">
<p>The situation is analogous for <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/test">userland/test</a>, <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-executables-in-tree">test-executables-in-tree</a> and <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-executables">test-executables</a>, which are further documented at: <a href="#user-mode-tests">Section 11.2, &#8220;User mode tests&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Do a more clean out-of-tree build instead and run the program:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --gcc-which host --userland-build-id host
./run --emulator native --userland userland/c/hello.c --userland-build-id host</pre>
</div>
</div>
<div class="paragraph">
<p>Here we:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>put the host executables in a separate <a href="#build-variants">build variant</a> to avoid conflict with Buildroot builds.</p>
</li>
<li>
<p>ran with the <code>--emulator native</code> option to run the program natively</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In this case you can debub the program with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm --emulator native --userland userland/c/hello.c --userland-build-id host</pre>
</div>
</div>
<div class="paragraph">
<p>as shown at: <a href="#debug-the-emulator">Section 23.8, &#8220;Debug the emulator&#8221;</a>, although direct GDB host usage works as well of course.</p>
</div>
</div>
<div class="sect4">
<h5 id="userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode"><a class="anchor" href="#userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode"></a><a class="link" href="#userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode">2.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode</a></h5>
<div class="paragraph">
<p>If you are lazy to built the Buildroot toolchain and QEMU, but want to run e.g. ARM <a href="#userland-assembly">Userland assembly</a> in <a href="#user-mode-simulation">User mode simulation</a>, you can get away on Ubuntu 18.04 with just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
./build-userland \
  --arch aarch64 \
  --gcc-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>--gcc-which host</code>: use the host toolchain.</p>
<div class="paragraph">
<p>We must pass this to <code>./run</code> as well because QEMU must know which dynamic libraries to use. See also: <a href="#user-mode-static-executables">Section 11.5, &#8220;User mode static executables&#8221;</a>.</p>
</div>
</li>
<li>
<p><code>--userland-build-id host</code>: put the host built into a <a href="#build-variants">Build variants</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This present the usual trade-offs of using prebuilts as mentioned at: <a href="#prebuilt">Section 2.6, &#8220;Prebuilt setup&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Other functionality are analogous, e.g. testing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables \
  --arch aarch64 \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and <a href="#user-mode-gdb">User mode GDB</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --gdb \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="userland-setup-getting-started-full-system"><a class="anchor" href="#userland-setup-getting-started-full-system"></a><a class="link" href="#userland-setup-getting-started-full-system">2.8.2.3. Userland setup getting started full system</a></h5>
<div class="paragraph">
<p>First ensure that <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a> is working.</p>
</div>
<div class="paragraph">
<p>After doing that setup, you can already execute your userland programs from inside QEMU: the only missing step is how to rebuild executables and run them.</p>
</div>
<div class="paragraph">
<p>And the answer is exactly analogous to what is shown at: <a href="#your-first-kernel-module-hack">Section 2.2.2.2, &#8220;Your first kernel module hack&#8221;</a></p>
</div>
<div class="paragraph">
<p>For example, if we modify <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a> to print out something different, we can just rebuild it with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-userland">build-userland</a>. <code>./build</code> calls that script automatically for us when doing the initial full build.</p>
</div>
<div class="paragraph">
<p>Now, run the program either without rebooting use the <a href="#9p">9P</a> mount:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/mnt/9p/out_rootfs_overlay/c/hello.out</pre>
</div>
</div>
<div class="paragraph">
<p>or shutdown QEMU, add the executable to the root filesystem:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot</pre>
</div>
</div>
<div class="paragraph">
<p>reboot and use the root filesystem as usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./hello.out</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-setup"><a class="anchor" href="#baremetal-setup"></a><a class="link" href="#baremetal-setup">2.9. Baremetal setup</a></h3>
<div class="sect3">
<h4 id="about-the-baremetal-setup"><a class="anchor" href="#about-the-baremetal-setup"></a><a class="link" href="#about-the-baremetal-setup">2.9.1. About the baremetal setup</a></h4>
<div class="paragraph">
<p>This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.</p>
</div>
<div class="paragraph">
<p><code>x86_64</code> is not currently supported, only <code>arm</code> and <code>aarch64</code>: I had made some x86 bare metal examples at: <a href="https://github.com/cirosantilli/x86-bare-metal-examples" class="bare">https://github.com/cirosantilli/x86-bare-metal-examples</a> but I&#8217;m lazy to port them here now. Pull requests are welcome.</p>
</div>
<div class="paragraph">
<p>The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.</p>
</div>
<div class="paragraph">
<p>This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.</p>
</div>
<div class="paragraph">
<p>You can also use C and a subset of the C standard library because we enable <a href="https://en.wikipedia.org/wiki/Newlib">Newlib</a> by default. See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077" class="bare">https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/13063055/does-a-libc-os-exist/59771531#59771531" class="bare">https://stackoverflow.com/questions/13063055/does-a-libc-os-exist/59771531#59771531</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Our C bare-metal compiler is built with <a href="https://github.com/crosstool-ng/crosstool-ng">crosstool-NG</a>. If you have already built <a href="#qemu-buildroot-setup">Buildroot</a> previously, you will end up with two GCCs installed. Unfortunately I don&#8217;t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: <a href="https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869" class="bare">https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869</a></p>
</div>
</div>
<div class="sect3">
<h4 id="baremetal-setup-getting-started"><a class="anchor" href="#baremetal-setup-getting-started"></a><a class="link" href="#baremetal-setup-getting-started">2.9.2. Baremetal setup getting started</a></h4>
<div class="paragraph">
<p>Every <code>.c</code> file inside <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/">baremetal/</a> and <code>.S</code> file inside <code>baremetal/arch/&lt;arch&gt;/</code> generates a separate baremetal image.</p>
</div>
<div class="paragraph">
<p>For example, to run <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/dump_regs.c">baremetal/arch/aarch64/dump_regs.c</a> in QEMU do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --arch aarch64 --download-dependencies qemu-baremetal
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c</pre>
</div>
</div>
<div class="paragraph">
<p>And the terminal prints the values of certain system registers. This example prints registers that are only accessible from <a href="#arm-exception-levels">EL1</a> or higher, and thus could not be run in userland.</p>
</div>
<div class="paragraph">
<p>In addition to the examples under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/">baremetal/</a>, several of the <a href="#userland-content">userland examples</a> can also be run in baremetal! This is largely due to the <a href="#about-the-baremetal-setup">awesomeness of Newlib</a>.</p>
</div>
<div class="paragraph">
<p>The examples that work include most <a href="#c">C examples</a> that don&#8217;t rely on complicated syscalls such as threads, and almost all the <a href="#userland-assembly">Userland assembly</a> examples.</p>
</div>
<div class="paragraph">
<p>The exact list of userland programs that work in baremetal is specified in <a href="#path-properties">path_properties.py</a> with the <code>baremetal</code> property, but you can also easily find it out with a <a href="#baremetal-tests">baremetal test dry run</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables --arch aarch64 --dry-run --mode baremetal</pre>
</div>
</div>
<div class="paragraph">
<p>For example, we can run the C hello world <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a> simply as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>and that outputs to the serial port the string:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>hello</pre>
</div>
</div>
<div class="paragraph">
<p>which QEMU shows on the host terminal.</p>
</div>
<div class="paragraph">
<p>To modify a baremetal program, simply edit the file, e.g.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>and rebuild:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-baremetal --arch aarch64
./run --arch aarch64 --baremetal userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p><code>./build qemu-baremetal</code> that we run previously is only needed for the initial build. That script calls <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-baremetal">build-baremetal</a> for us, in addition to building prerequisites such as QEMU and crosstool-NG.</p>
</div>
<div class="paragraph">
<p><code>./build-baremetal</code> uses crosstool-NG, and so it must be preceded by <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-crosstool-ng">build-crosstool-ng</a>, which <code>./build qemu-baremetal</code> also calls.</p>
</div>
<div class="paragraph">
<p>Now let&#8217;s run <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/add.S">userland/arch/aarch64/add.S</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/arch/aarch64/add.S</pre>
</div>
</div>
<div class="paragraph">
<p>This time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.</p>
</div>
<div class="paragraph">
<p>You can see a sample assertion fail in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/assert_fail.c">userland/c/assert_fail.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/c/assert_fail.c</pre>
</div>
</div>
<div class="paragraph">
<p>and the terminal contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lkmc_exit_status_134
error: simulation error detected by parsing logs</pre>
</div>
</div>
<div class="paragraph">
<p>and the exit status of our script is 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>You can run all the baremetal examples in one go and check that all assertions passed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables --arch aarch64 --mode baremetal</pre>
</div>
</div>
<div class="paragraph">
<p>To use gem5 instead of QEMU do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --download-dependencies gem5-baremetal
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>and then <a href="#qemu-buildroot-setup">as usual</a> open a shell with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-shell</pre>
</div>
</div>
<div class="paragraph">
<p>Or as usual, <a href="#tmux">tmux</a> users can do both in one go with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --tmux</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: the carriage returns are a bit different than in QEMU, see: <a href="#gem5-baremetal-carriage-return">Section 33.6, &#8220;gem5 baremetal carriage return&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Note that <code>./build-baremetal</code> requires the <code>--emulator gem5</code> option, and generates separate executable images for both, as can be seen from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator qemu image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 image)"</pre>
</div>
</div>
<div class="paragraph">
<p>This is unlike the Linux kernel that has a single image for both QEMU and gem5:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo "$(./getvar --arch aarch64 --emulator qemu image)"
echo "$(./getvar --arch aarch64 --emulator gem5 image)"</pre>
</div>
</div>
<div class="paragraph">
<p>The reason for that is that on baremetal we don&#8217;t parse the <a href="#device-tree">device tress</a> from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.</p>
</div>
<div class="paragraph">
<p><code>gem5</code> also supports the <code>RealViewPBX</code> machine, which represents an older hardware compared to the default <code>VExpress_GEM5_V1</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX</pre>
</div>
</div>
<div class="paragraph">
<p>see also: <a href="#gem5-arm-platforms">Section 24.18, &#8220;gem5 ARM platforms&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>This generates yet new separate images with new magic constants:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine VExpress_GEM5_V1 image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX      image)"</pre>
</div>
</div>
<div class="paragraph">
<p>But just stick to newer and better <code>VExpress_GEM5_V1</code> unless you have a good reason to use <code>RealViewPBX</code>.</p>
</div>
<div class="paragraph">
<p>When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: <a href="#userland-assembly">Section 28, &#8220;Userland assembly&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>For more information on baremetal, see the section: <a href="#baremetal">Section 33, &#8220;Baremetal&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The following subjects are particularly important:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#tracing">Tracing</a></p>
</li>
<li>
<p><a href="#baremetal-gdb-step-debug">Baremetal GDB step debug</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="build-the-documentation"><a class="anchor" href="#build-the-documentation"></a><a class="link" href="#build-the-documentation">2.10. Build the documentation</a></h3>
<div class="paragraph">
<p>You don&#8217;t need to depend on GitHub.</p>
</div>
<div class="paragraph">
<p>For a quick and dirty build, install <a href="https://asciidoctor.org/">Asciidoctor</a> however you like and build:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>asciidoctor README.adoc
xdg-open README.html</pre>
</div>
</div>
<div class="paragraph">
<p>For development, you will want to do a more controlled build with extra error checking as follows.</p>
</div>
<div class="paragraph">
<p>For the initial build do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./setup
./build --download-dependencies docs</pre>
</div>
</div>
<div class="paragraph">
<p>which also downloads build dependencies.</p>
</div>
<div class="paragraph">
<p>Then the following times just to the faster:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-doc</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-doc">build-doc</a></p>
</div>
<div class="paragraph">
<p>The HTML output is located at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>xdg-open out/README.html</pre>
</div>
</div>
<div class="paragraph">
<p>More information about our documentation internals can be found at: <a href="#documentation">Section 38.5, &#8220;Documentation&#8221;</a></p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="gdb"><a class="anchor" href="#gdb"></a><a class="link" href="#gdb">3. GDB step debug</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="gdb-step-debug-kernel-boot"><a class="anchor" href="#gdb-step-debug-kernel-boot"></a><a class="link" href="#gdb-step-debug-kernel-boot">3.1. GDB step debug kernel boot</a></h3>
<div class="paragraph">
<p><code>--gdb-wait</code> makes QEMU and gem5 wait for a GDB connection, otherwise we could accidentally go past the point we want to break at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>Say you want to break at <code>start_kernel</code>. So on another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>or at a given line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb init/main.c:1088</pre>
</div>
</div>
<div class="paragraph">
<p>Now QEMU will stop there, and you can use the normal GDB commands:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>list
next
continue</pre>
</div>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642" class="bare">https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607" class="bare">https://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gdb-step-debug-kernel-boot-other-archs"><a class="anchor" href="#gdb-step-debug-kernel-boot-other-archs"></a><a class="link" href="#gdb-step-debug-kernel-boot-other-archs">3.1.1. GDB step debug kernel boot other archs</a></h4>
<div class="paragraph">
<p>Just don&#8217;t forget to pass <code>--arch</code> to <code>./run-gdb</code>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch aarch64 start_kernel</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="kernel-o0"><a class="anchor" href="#kernel-o0"></a><a class="link" href="#kernel-o0">3.1.2. Disable kernel compiler optimizations</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0" class="bare">https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0</a></p>
</div>
<div class="paragraph">
<p><code>O=0</code> is an impossible dream, <code>O=2</code> being the default.</p>
</div>
<div class="paragraph">
<p>So get ready for some weird jumps, and <code>&lt;value optimized out&gt;</code> fun. Why, Linux, why.</p>
</div>
<div class="paragraph">
<p>The <code>-O</code> level of some other userland content can be controlled as explained at: <a href="#optimization-level-of-a-build">Optimization level of a build</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdb-step-debug-kernel-post-boot"><a class="anchor" href="#gdb-step-debug-kernel-post-boot"></a><a class="link" href="#gdb-step-debug-kernel-post-boot">3.2. GDB step debug kernel post-boot</a></h3>
<div class="paragraph">
<p>Let&#8217;s observe the kernel <code>write</code> system call as it reacts to some userland actions.</p>
</div>
<div class="paragraph">
<p>Start QEMU with just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>and after boot inside a shell run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./count.sh</pre>
</div>
</div>
<div class="paragraph">
<p>which counts to infinity to stdout. Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/count.sh">rootfs_overlay/lkmc/count.sh</a>.</p>
</div>
<div class="paragraph">
<p>Then in another shell, run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb</pre>
</div>
</div>
<div class="paragraph">
<p>and then hit:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-C
break __x64_sys_write
continue
continue
continue</pre>
</div>
</div>
<div class="paragraph">
<p>And you now control the counting on the first shell from GDB!</p>
</div>
<div class="paragraph">
<p>Before v4.17, the symbol name was just <code>sys_write</code>, the change happened at <a href="https://github.com/torvalds/linux/commit/d5a00528b58cdb2c71206e18bd021e34c4eab878">d5a00528b58cdb2c71206e18bd021e34c4eab878</a>. As of Linux v 4.19, the function is called <code>sys_write</code> in <code>arm</code>, and <code>__arm64_sys_write</code> in <code>aarch64</code>. One good way to find it if the name changes again is to try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rbreak .*sys_write</pre>
</div>
</div>
<div class="paragraph">
<p>or just have a quick look at the sources!</p>
</div>
<div class="paragraph">
<p>When you hit <code>Ctrl-C</code>, if we happen to be inside kernel code at that point, which is very likely if there are no heavy background tasks waiting, and we are just waiting on a <code>sleep</code> type system call of the command prompt, we can already see the source for the random place inside the kernel where we stopped.</p>
</div>
</div>
<div class="sect2">
<h3 id="tmux"><a class="anchor" href="#tmux"></a><a class="link" href="#tmux">3.3. tmux</a></h3>
<div class="paragraph">
<p>tmux just makes things even more fun by allowing us to see both the terminal for:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>emulator stdout</p>
</li>
<li>
<p><a href="#gdb">GDB step debug</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>at once without dragging windows around!</p>
</div>
<div class="paragraph">
<p>First start <code>tmux</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>tmux</pre>
</div>
</div>
<div class="paragraph">
<p>Now that you are inside a shell inside tmux, you can start GDB simply with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb</pre>
</div>
</div>
<div class="paragraph">
<p>which is just a convenient shortcut for:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait --tmux --tmux-args start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>This splits the terminal into two panes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>left: usual QEMU with terminal</p>
</li>
<li>
<p>right: GDB</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and focuses on the GDB pane.</p>
</div>
<div class="paragraph">
<p>Now you can navigate with the usual tmux shortcuts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>switch between the two panes with: <code>Ctrl-B O</code></p>
</li>
<li>
<p>close either pane by killing its terminal with <code>Ctrl-D</code> as usual</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>See the tmux manual for further details:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>man tmux</pre>
</div>
</div>
<div class="paragraph">
<p>To start again, switch back to the QEMU pane with <code>Ctrl-O</code>, kill the emulator, and re-run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb</pre>
</div>
</div>
<div class="paragraph">
<p>This automatically clears the GDB pane, and starts a new one.</p>
</div>
<div class="paragraph">
<p>The option <code>--tmux-args</code> determines which options will be passed to the program running on the second tmux pane, and is equivalent to:</p>
</div>
<div class="paragraph">
<p>This is equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait
./run-gdb start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>Due to Python&#8217;s CLI parsing quicks, if the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run-gdb">run-gdb</a> arguments start with a dash <code>-</code>, you have to use the <code>=</code> sign, e.g. to <a href="#gdb-step-debug-early-boot">GDB step debug early boot</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb --tmux-args=--no-continue</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://unix.stackexchange.com/questions/152738/how-to-split-a-new-window-and-run-a-command-in-this-new-window-using-tmux/432111#432111" class="bare">https://unix.stackexchange.com/questions/152738/how-to-split-a-new-window-and-run-a-command-in-this-new-window-using-tmux/432111#432111</a></p>
</div>
<div class="sect3">
<h4 id="tmux-gem5"><a class="anchor" href="#tmux-gem5"></a><a class="link" href="#tmux-gem5">3.3.1. tmux gem5</a></h4>
<div class="paragraph">
<p>If you are using gem5 instead of QEMU, <code>--tmux</code> has a different effect by default: it opens the gem5 terminal instead of the debugger:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --tmux</pre>
</div>
</div>
<div class="paragraph">
<p>To open a new pane with GDB instead of the terminal, use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb</pre>
</div>
</div>
<div class="paragraph">
<p>which is equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --gdb-wait --tmux --tmux-args start_kernel --tmux-program gdb</pre>
</div>
</div>
<div class="paragraph">
<p><code>--tmux-program</code> implies <code>--tmux</code>, so we can just write:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --gdb-wait --tmux-program gdb</pre>
</div>
</div>
<div class="paragraph">
<p>If you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with <code>./gem5-shell</code>.</p>
</div>
<div class="paragraph">
<p>From inside tmux, you can create new terminals on a new window with <code>Ctrl-B C</code> split a pane yet again vertically with <code>Ctrl-B %</code> or horizontally with <code>Ctrl-B "</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdb-step-debug-kernel-module"><a class="anchor" href="#gdb-step-debug-kernel-module"></a><a class="link" href="#gdb-step-debug-kernel-module">3.4. GDB step debug kernel module</a></h3>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831" class="bare">https://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831</a></p>
</div>
<div class="paragraph">
<p>Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.</p>
</div>
<div class="paragraph">
<p>So we cannot set the breakpoints before <code>insmod</code>.</p>
</div>
<div class="paragraph">
<p>However, the Linux kernel GDB scripts offer the <code>lx-symbols</code> command, which takes care of that beautifully for us.</p>
</div>
<div class="paragraph">
<p>Shell 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>Wait for the boot to end and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod timer.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/timer.c">kernel_modules/timer.c</a>.</p>
</div>
<div class="paragraph">
<p>This prints a message to dmesg every second.</p>
</div>
<div class="paragraph">
<p>Shell 2:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb</pre>
</div>
</div>
<div class="paragraph">
<p>In GDB, hit <code>Ctrl-C</code>, and note how it says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>scanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules
loading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko</pre>
</div>
</div>
<div class="paragraph">
<p>That&#8217;s <code>lx-symbols</code> working! Now simply:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>break lkmc_timer_callback
continue
continue
continue</pre>
</div>
</div>
<div class="paragraph">
<p>and we now control the callback from GDB!</p>
</div>
<div class="paragraph">
<p>Just don&#8217;t forget to remove your breakpoints after <code>rmmod</code>, or they will point to stale memory locations.</p>
</div>
<div class="paragraph">
<p>TODO: why does <code>break work_func</code> for <code>insmod kthread.ko</code> not very well? Sometimes it breaks but not others.</p>
</div>
<div class="sect3">
<h4 id="gdb-step-debug-kernel-module-arm"><a class="anchor" href="#gdb-step-debug-kernel-module-arm"></a><a class="link" href="#gdb-step-debug-kernel-module-arm">3.4.1. GDB step debug kernel module insmodded by init on ARM</a></h4>
<div class="paragraph">
<p>TODO on <code>arm</code> 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and <code>lx-symbols</code> fails with the message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>loading vmlinux
Traceback (most recent call last):
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 163, in invoke
    self.load_all_symbols()
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 150, in load_all_symbols
    [self.load_module_symbols(module) for module in module_list]
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 110, in load_module_symbols
    module_name = module['name'].string()
gdb.MemoryError: Cannot access memory at address 0xbf0000cc
Error occurred in Python command: Cannot access memory at address 0xbf0000cc</pre>
</div>
</div>
<div class="paragraph">
<p>Can&#8217;t reproduce on <code>x86_64</code> and <code>aarch64</code> are fine.</p>
</div>
<div class="paragraph">
<p>It is kind of random: if you just <code>insmod</code> manually and then immediately <code>./run-gdb --arch arm</code>, then it usually works.</p>
</div>
<div class="paragraph">
<p>But this fails most of the time: shell 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --eval-after 'insmod hello.ko'</pre>
</div>
</div>
<div class="paragraph">
<p>shell 2:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm</pre>
</div>
</div>
<div class="paragraph">
<p>then hit <code>Ctrl-C</code> on shell 2, and voila.</p>
</div>
<div class="paragraph">
<p>Then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/modules</pre>
</div>
</div>
<div class="paragraph">
<p>says that the load address is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0xbf000000</pre>
</div>
</div>
<div class="paragraph">
<p>so it is close to the failing <code>0xbf0000cc</code>.</p>
</div>
<div class="paragraph">
<p><code>readelf</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- -s "$(./getvar kernel_modules_build_subdir)/hello.ko"</pre>
</div>
</div>
<div class="paragraph">
<p>does not give any interesting hits at <code>cc</code>, no symbol was placed that far.</p>
</div>
</div>
<div class="sect3">
<h4 id="gdb-module-init"><a class="anchor" href="#gdb-module-init"></a><a class="link" href="#gdb-module-init">3.4.2. GDB module_init</a></h4>
<div class="paragraph">
<p>TODO find a more convenient method. We have working methods, but they are not ideal.</p>
</div>
<div class="paragraph">
<p>This is not very easy, since by the time the module finishes loading, and <code>lx-symbols</code> can work properly, <code>module_init</code> has already finished running!</p>
</div>
<div class="paragraph">
<p>Possibly asked at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded" class="bare">https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module" class="bare">https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="gdb-module-init-step-into-it"><a class="anchor" href="#gdb-module-init-step-into-it"></a><a class="link" href="#gdb-module-init-step-into-it">3.4.2.1. GDB module_init step into it</a></h5>
<div class="paragraph">
<p>This is the best method we&#8217;ve found so far.</p>
</div>
<div class="paragraph">
<p>The kernel calls <code>module_init</code> synchronously, therefore it is not hard to step into that call.</p>
</div>
<div class="paragraph">
<p>As of 4.16, the call happens in <code>do_one_initcall</code>, so we can do in shell 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>shell 2 after boot finishes (because there are other calls to <code>do_init_module</code> at boot, presumably for the built-in modules):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb do_one_initcall</pre>
</div>
</div>
<div class="paragraph">
<p>then step until the line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>833         ret = fn();</pre>
</div>
</div>
<div class="paragraph">
<p>which does the actual call, and then step into it.</p>
</div>
<div class="paragraph">
<p>For the next time, you can also put a breakpoint there directly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb init/main.c:833</pre>
</div>
</div>
<div class="paragraph">
<p>How we found this out: first we got <a href="#gdb-module-init-calculate-entry-address">GDB module_init calculate entry address</a> working, and then we did a <code>bt</code>. AKA cheating :-)</p>
</div>
</div>
<div class="sect4">
<h5 id="gdb-module-init-calculate-entry-address"><a class="anchor" href="#gdb-module-init-calculate-entry-address"></a><a class="link" href="#gdb-module-init-calculate-entry-address">3.4.2.2. GDB module_init calculate entry address</a></h5>
<div class="paragraph">
<p>This works, but is a bit annoying.</p>
</div>
<div class="paragraph">
<p>The key observation is that the load address of kernel modules is deterministic: there is a pre allocated memory region <a href="https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt" class="bare">https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt</a> "module mapping space" filled from bottom up.</p>
</div>
<div class="paragraph">
<p>So once we find the address the first time, we can just reuse it afterwards, as long as we don&#8217;t modify the module.</p>
</div>
<div class="paragraph">
<p>Do a fresh boot and get the module:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './pr_debug.sh;insmod fops.ko;./linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>The boot must be fresh, because the load address changes every time we insert, even after removing previous modules.</p>
</div>
<div class="paragraph">
<p>The base address shows on terminal:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0xffffffffc0000000 .text</pre>
</div>
</div>
<div class="paragraph">
<p>Now let&#8217;s find the offset of <code>myinit</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- \
  -s "$(./getvar kernel_modules_build_subdir)/fops.ko" | \
  grep myinit</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    30: 0000000000000240    43 FUNC    LOCAL  DEFAULT    2 myinit</pre>
</div>
</div>
<div class="paragraph">
<p>so the offset address is <code>0x240</code> and we deduce that the function will be placed at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0xffffffffc0000000 + 0x240 = 0xffffffffc0000240</pre>
</div>
</div>
<div class="paragraph">
<p>Now we can just do a fresh boot on shell 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'insmod fops.ko;./linux/poweroff.out' --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>and on shell 2:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb '*0xffffffffc0000240'</pre>
</div>
</div>
<div class="paragraph">
<p>GDB then breaks, and <code>lx-symbols</code> works.</p>
</div>
</div>
<div class="sect4">
<h5 id="gdb-module-init-break-at-the-end-of-sys-init-module"><a class="anchor" href="#gdb-module-init-break-at-the-end-of-sys-init-module"></a><a class="link" href="#gdb-module-init-break-at-the-end-of-sys-init-module">3.4.2.3. GDB module_init break at the end of sys_init_module</a></h5>
<div class="paragraph">
<p>TODO not working. This could be potentially very convenient.</p>
</div>
<div class="paragraph">
<p>The idea here is to break at a point late enough inside <code>sys_init_module</code>, at which point <code>lx-symbols</code> can be called and do its magic.</p>
</div>
<div class="paragraph">
<p>Beware that there are both <code>sys_init_module</code> and <code>sys_finit_module</code> syscalls, and <code>insmod</code> uses <code>fmodule_init</code> by default.</p>
</div>
<div class="paragraph">
<p>Both call <code>do_module_init</code> however, which is what <code>lx-symbols</code> hooks to.</p>
</div>
<div class="paragraph">
<p>If we try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>b sys_finit_module</pre>
</div>
</div>
<div class="paragraph">
<p>then hitting:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>n</pre>
</div>
</div>
<div class="paragraph">
<p>does not break, and insertion happens, likely because of optimizations? <a href="#kernel-o0">Disable kernel compiler optimizations</a></p>
</div>
<div class="paragraph">
<p>Then we try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>b do_init_module</pre>
</div>
</div>
<div class="paragraph">
<p>A naive:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fin</pre>
</div>
</div>
<div class="paragraph">
<p>also fails to break!</p>
</div>
<div class="paragraph">
<p>Finally, in despair we notice that <a href="#pr-debug">pr_debug</a> prints the kernel load address as explained at <a href="#bypass-lx-symbols">Bypass lx-symbols</a>.</p>
</div>
<div class="paragraph">
<p>So, if we set a breakpoint just after that message is printed by searching where that happens on the Linux source code, we must be able to get the correct load address before <code>init_module</code> happens.</p>
</div>
</div>
<div class="sect4">
<h5 id="gdb-module-init-add-trap-instruction"><a class="anchor" href="#gdb-module-init-add-trap-instruction"></a><a class="link" href="#gdb-module-init-add-trap-instruction">3.4.2.4. GDB module_init add trap instruction</a></h5>
<div class="paragraph">
<p>This is another possibility: we could modify the module source by adding a trap instruction of some kind.</p>
</div>
<div class="paragraph">
<p>This appears to be described at: <a href="https://www.linuxjournal.com/article/4525" class="bare">https://www.linuxjournal.com/article/4525</a></p>
</div>
<div class="paragraph">
<p>But it refers to a <code>gdbstart</code> script which is not in the tree anymore and beyond my <code>git log</code> capabilities.</p>
</div>
<div class="paragraph">
<p>And just adding:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>asm( " int $3");</pre>
</div>
</div>
<div class="paragraph">
<p>directly gives an <a href="#oops">oops</a> as I&#8217;d expect.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="bypass-lx-symbols"><a class="anchor" href="#bypass-lx-symbols"></a><a class="link" href="#bypass-lx-symbols">3.4.3. Bypass lx-symbols</a></h4>
<div class="paragraph">
<p>Useless, but a good way to show how hardcore you are. Disable <code>lx-symbols</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --no-lxsymbols</pre>
</div>
</div>
<div class="paragraph">
<p>From inside guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod timer.ko
cat /proc/modules</pre>
</div>
</div>
<div class="paragraph">
<p>as mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818" class="bare">https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module" class="bare">https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This will give a line of form:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fops 2327 0 - Live 0xfffffffa00000000</pre>
</div>
</div>
<div class="paragraph">
<p>And then tell GDB where the module was loaded with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-C
add-symbol-file ../../../rootfs_overlay/x86_64/timer.ko 0xffffffffc0000000
0xffffffffc0000000</pre>
</div>
</div>
<div class="paragraph">
<p>Alternatively, if the module panics before you can read <code>/proc/modules</code>, there is a <a href="#pr-debug">pr_debug</a> which shows the load address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 8 &gt; /proc/sys/kernel/printk
echo 'file kernel/module.c +p' &gt; /sys/kernel/debug/dynamic_debug/control
./linux/myinsmod.out hello.ko</pre>
</div>
</div>
<div class="paragraph">
<p>And then search for a line of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[   84.877482]  0xfffffffa00000000 .text</pre>
</div>
</div>
<div class="paragraph">
<p>Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdb-step-debug-early-boot"><a class="anchor" href="#gdb-step-debug-early-boot"></a><a class="link" href="#gdb-step-debug-early-boot">3.5. GDB step debug early boot</a></h3>
<div class="paragraph">
<p>TODO successfully debug the very first instruction that the Linux kernel runs, before <code>start_kernel</code>!</p>
</div>
<div class="paragraph">
<p>Break at the very first instruction executed by QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --no-continue</pre>
</div>
</div>
<div class="paragraph">
<p>Note however that early boot parts appear to be relocated in memory somehow, and therefore:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>you won&#8217;t see the source location in GDB, only assembly</p>
</li>
<li>
<p>you won&#8217;t be able to break by symbol in those early locations</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Further discussion at: <a href="#linux-kernel-entry-point">Linux kernel entry point</a>.</p>
</div>
<div class="paragraph">
<p>In the specific case of gem5 aarch64 at least:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>gem5 relocates the kernel in memory to a fixed location, see e.g. <a href="https://gem5.atlassian.net/browse/GEM5-787" class="bare">https://gem5.atlassian.net/browse/GEM5-787</a></p>
</li>
<li>
<p><code>--param 'system.workload.early_kernel_symbols=True</code> should in theory duplicate the symbols to the correct physical location, but it was broken at one point: <a href="https://gem5.atlassian.net/browse/GEM5-785" class="bare">https://gem5.atlassian.net/browse/GEM5-785</a></p>
</li>
<li>
<p>gem5 executes directly from vmlinux, so there is no decompression code involved, so you actually immediately start running the "true" first instruction from <code>head.S</code> as described at: <a href="https://stackoverflow.com/questions/18266063/does-linux-kernel-have-main-function/33422401#33422401" class="bare">https://stackoverflow.com/questions/18266063/does-linux-kernel-have-main-function/33422401#33422401</a></p>
</li>
<li>
<p>once the MMU gets turned on at kernel symbol <code>__primary_switched</code>, the virtual address matches the ELF symbols, and you start seeing correct symbols without the need for <code>early_kernel_symbols</code>. This can be observed clearly with <code>function_trace = True</code>: <a href="https://stackoverflow.com/questions/64049487/how-to-trace-executed-guest-function-symbol-names-with-their-timestamp-in-gem5/64049488#64049488" class="bare">https://stackoverflow.com/questions/64049487/how-to-trace-executed-guest-function-symbol-names-with-their-timestamp-in-gem5/64049488#64049488</a> which produces:</p>
<div class="literalblock">
<div class="content">
<pre>0: _kernel_flags_le_lo32 (12500)
12500: __crc_tcp_add_backlog (1000)
13500: __crc_crypto_alg_tested (6500)
20000: __crc_tcp_add_backlog (10000)
30000: __crc_crypto_alg_tested (500)
30500: __crc_scsi_is_host_device (5000)
35500: __crc_crypto_alg_tested (1500)
37000: __crc_scsi_is_host_device (4000)
41000: __crc_crypto_alg_tested (3000)
44000: __crc_tcp_add_backlog (263500)
307500: __crc_crypto_alg_tested (975500)
1283000: __crc_tcp_add_backlog (77191500)
78474500: __crc_crypto_alg_tested (1000)
78475500: __crc_scsi_is_host_device (19500)
78495000: __crc_crypto_alg_tested (500)
78495500: __crc_scsi_is_host_device (13500)
78509000: __primary_switched (14000)
78523000: memset (21118000)
99641000: __primary_switched (2500)
99643500: start_kernel (11000)</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that <code><em>primary_switched</code> is the first non-trash symbol (non-<code></em>crc_*</code> and non-<code><em>kernel_flags</em>*</code>, which are just informative symbols, not actual executable code)</p>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="linux-kernel-entry-point"><a class="anchor" href="#linux-kernel-entry-point"></a><a class="link" href="#linux-kernel-entry-point">3.5.1. Linux kernel entry point</a></h4>
<div class="paragraph">
<p>TODO <a href="https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot" class="bare">https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot</a></p>
</div>
<div class="paragraph">
<p>As mentioned at: <a href="#gdb-step-debug-early-boot">GDB step debug early boot</a>, the very first kernel instructions executed appear to be placed into memory at a different location than that of the kernel ELF section.</p>
</div>
<div class="paragraph">
<p>As a result, we are unable to break on early symbols such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb extract_kernel
./run-gdb main</pre>
</div>
</div>
<div class="paragraph">
<p><a href="#gem5-execall-trace-format">gem5 ExecAll trace format</a>&gt;&gt; however does show the right symbols however! This could be because <a href="#vmlinux-vs-bzimage-vs-zimage-vs-image">gem5 uses vmlinux to boot</a>, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.</p>
</div>
<div class="paragraph">
<p>I also tried to hack <code>run-gdb</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>@@ -81,7 +81,7 @@ else
 ${gdb} \
 -q \\
 -ex 'add-auto-load-safe-path $(pwd)' \\
--ex 'file vmlinux' \\
+-ex 'file arch/arm/boot/compressed/vmlinux' \\
 -ex 'target remote localhost:${port}' \\
 ${brk} \
 -ex 'continue' \\</pre>
</div>
</div>
<div class="paragraph">
<p>and no I do have the symbols from <code>arch/arm/boot/compressed/vmlinux'</code>, but the breaks still don&#8217;t work.</p>
</div>
<div class="paragraph">
<p>v4.19 also added a <code>CONFIG_HAVE_KERNEL_UNCOMPRESSED=y</code> option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. <code>aarch64</code> however is already uncompressed by default, so might be the easiest one. See also: <a href="#vmlinux-vs-bzimage-vs-zimage-vs-image">Section 17.20.1, &#8220;vmlinux vs bzImage vs zImage vs Image&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>You then need the associated <code>KERNEL_UNCOMPRESSED</code> to enable it if available:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>config KERNEL_UNCOMPRESSED
    bool "None"
    depends on HAVE_KERNEL_UNCOMPRESSED</pre>
</div>
</div>
<div class="sect4">
<h5 id="arm64-secondary-cpu-entry-point"><a class="anchor" href="#arm64-secondary-cpu-entry-point"></a><a class="link" href="#arm64-secondary-cpu-entry-point">3.5.1.1. arm64 secondary CPU entry point</a></h5>
<div class="paragraph">
<p>In gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be <code>secondary_holding_pen</code> as shown at <a href="https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90" class="bare">https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90</a></p>
</div>
<div class="paragraph">
<p>What happens is that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the bootloader goes in in WFE</p>
</li>
<li>
<p>the kernel writes the entry point to the secondary CPU (the address of <code>secondary_holding_pen</code>) with CPU0 at the address given to the kernel in the <code>cpu-release-addr</code> of the DTB</p>
</li>
<li>
<p>the kernel wakes up the bootloader with a SEV, and the bootloader boots to the address the kernel told it</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The CPU0 action happens at: <a href="https://github.com/cirosantilli/linux/blob/v5.7/arch/arm64/kernel/smp_spin_table.c" class="bare">https://github.com/cirosantilli/linux/blob/v5.7/arch/arm64/kernel/smp_spin_table.c</a>:</p>
</div>
<div class="paragraph">
<p>Here&#8217;s the code that writes the address and does SEV:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static int smp_spin_table_cpu_prepare(unsigned int cpu)
{
	__le64 __iomem *release_addr;

	if (!cpu_release_addr[cpu])
		return -ENODEV;

	/*
	 * The cpu-release-addr may or may not be inside the linear mapping.
	 * As ioremap_cache will either give us a new mapping or reuse the
	 * existing linear mapping, we can use it to cover both cases. In
	 * either case the memory will be MT_NORMAL.
	 */
	release_addr = ioremap_cache(cpu_release_addr[cpu],
				     sizeof(*release_addr));
	if (!release_addr)
		return -ENOMEM;

	/*
	 * We write the release address as LE regardless of the native
	 * endianess of the kernel. Therefore, any boot-loaders that
	 * read this address need to convert this address to the
	 * boot-loader's endianess before jumping. This is mandated by
	 * the boot protocol.
	 */
	writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);
	__flush_dcache_area((__force void *)release_addr,
			    sizeof(*release_addr));

	/*
	 * Send an event to wake up the secondary CPU.
	 */
	sev();</pre>
</div>
</div>
<div class="paragraph">
<p>and here&#8217;s the code that reads the value from the DTB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static int smp_spin_table_cpu_init(unsigned int cpu)
{
	struct device_node *dn;
	int ret;

	dn = of_get_cpu_node(cpu, NULL);
	if (!dn)
		return -ENODEV;

	/*
	 * Determine the address from which the CPU is polling.
	 */
	ret = of_property_read_u64(dn, "cpu-release-addr",
				   &amp;cpu_release_addr[cpu]);</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="linux-kernel-arch-agnostic-entry-point"><a class="anchor" href="#linux-kernel-arch-agnostic-entry-point"></a><a class="link" href="#linux-kernel-arch-agnostic-entry-point">3.5.2. Linux kernel arch-agnostic entry point</a></h4>
<div class="paragraph">
<p><code>start_kernel</code> is the first C function to be executed basically: <a href="https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401" class="bare">https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401</a></p>
</div>
<div class="paragraph">
<p>For the earlier arch-specific entry point, see: <a href="#linux-kernel-entry-point">Linux kernel entry point</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="linux-kernel-early-boot-messages"><a class="anchor" href="#linux-kernel-early-boot-messages"></a><a class="link" href="#linux-kernel-early-boot-messages">3.5.3. Linux kernel early boot messages</a></h4>
<div class="paragraph">
<p>When booting Linux on a slow emulator like <a href="#gem5">gem5</a>, what you observe is that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>first nothing shows for a while</p>
</li>
<li>
<p>then at once, a bunch of message lines show at once followed on aarch64 Linux 5.4.3 by:</p>
<div class="literalblock">
<div class="content">
<pre>[    0.081311] printk: console [ttyAMA0] enabled</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>This means of course that all the previous messages had been generated earlier and stored, but were only printed to the terminal once the terminal itself was enabled.</p>
</div>
<div class="paragraph">
<p>Notably for example the very first message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]</pre>
</div>
</div>
<div class="paragraph">
<p>happens very early in the boot process.</p>
</div>
<div class="paragraph">
<p>If you get a failure before that, it will be hard to see the print messages.</p>
</div>
<div class="paragraph">
<p>One possible solution is to parse the dmesg buffer, gem5 actually implements that: <a href="#gem5-m5out-system-dmesg-file">gem5 <code>m5out/system.workload.dmesg</code> file</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdb-step-debug-userland-processes"><a class="anchor" href="#gdb-step-debug-userland-processes"></a><a class="link" href="#gdb-step-debug-userland-processes">3.6. GDB step debug userland processes</a></h3>
<div class="paragraph">
<p>QEMU&#8217;s <code>-gdb</code> GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel" class="bare">https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb" class="bare">https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>You will generally want to use <a href="#gdbserver">gdbserver</a> for this as it is more reliable, but this method can overcome the following limitations of <code>gdbserver</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the emulator does not support host to guest networking. This seems to be the case for gem5 as explained at: <a href="#gem5-host-to-guest-networking">Section 15.3.1.3, &#8220;gem5 host to guest networking&#8221;</a></p>
</li>
<li>
<p>cannot see the start of the <code>init</code> process easily</p>
</li>
<li>
<p><code>gdbserver</code> alters the working of the kernel, and makes your run less representative</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Known limitations of direct userland debugging:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the kernel might switch context to another process or to the kernel itself e.g. on a system call, and then TODO confirm the PIC would go to weird places and source code would be missing.</p>
<div class="paragraph">
<p>Solutions to this are being researched at: <a href="#lx-ps">Section 3.10.1, &#8220;lx-ps&#8221;</a>.</p>
</div>
</li>
<li>
<p>TODO step into shared libraries. If I attempt to load them explicitly:</p>
<div class="literalblock">
<div class="content">
<pre>(gdb) sharedlibrary ../../staging/lib/libc.so.0
No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.</pre>
</div>
</div>
<div class="paragraph">
<p>since GDB does not know that libc is loaded.</p>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gdb-step-debug-userland-custom-init"><a class="anchor" href="#gdb-step-debug-userland-custom-init"></a><a class="link" href="#gdb-step-debug-userland-custom-init">3.6.1. GDB step debug userland custom init</a></h4>
<div class="paragraph">
<p>This is the userland debug setup most likely to work, since at init time there is only one userland executable running.</p>
</div>
<div class="paragraph">
<p>For executables from the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/">userland/</a> directory such as <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count.c">userland/posix/count.c</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Shell 1:</p>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait --kernel-cli 'init=/lkmc/posix/count.out'</pre>
</div>
</div>
</li>
<li>
<p>Shell 2:</p>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland userland/posix/count.c main</pre>
</div>
</div>
<div class="paragraph">
<p>Alternatively, we could also pass the full path to the executable:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar userland_build_dir)/posix/count.out" main</pre>
</div>
</div>
<div class="paragraph">
<p>Path resolution is analogous to <a href="#baremetal-setup-getting-started">that of <code>./run --baremetal</code></a>.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, as soon as boot ends, we are left inside a debug session that looks just like what <code>gdbserver</code> would produce.</p>
</div>
</div>
<div class="sect3">
<h4 id="gdb-step-debug-userland-busybox-init"><a class="anchor" href="#gdb-step-debug-userland-busybox-init"></a><a class="link" href="#gdb-step-debug-userland-busybox-init">3.6.2. GDB step debug userland BusyBox init</a></h4>
<div class="paragraph">
<p>BusyBox custom init process:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Shell 1:</p>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait --kernel-cli 'init=/bin/ls'</pre>
</div>
</div>
</li>
<li>
<p>Shell 2:</p>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>This follows BusyBox' convention of calling the main for each executable as <code>&lt;exec&gt;_main</code> since the <code>busybox</code> executable has many "mains".</p>
</div>
<div class="paragraph">
<p>BusyBox default init process:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Shell 1:</p>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait</pre>
</div>
</div>
</li>
<li>
<p>Shell 2:</p>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox init_main</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>init</code> cannot be debugged with <a href="#gdbserver">gdbserver</a> without modifying the source, or else <code>/sbin/init</code> exits early with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>"must be run as PID 1"</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gdb-step-debug-userland-non-init"><a class="anchor" href="#gdb-step-debug-userland-non-init"></a><a class="link" href="#gdb-step-debug-userland-non-init">3.6.3. GDB step debug userland non-init</a></h4>
<div class="paragraph">
<p>Non-init process:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Shell 1:</p>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait</pre>
</div>
</div>
</li>
<li>
<p>Shell 2:</p>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland userland/linux/rand_check.c main</pre>
</div>
</div>
</li>
<li>
<p>Shell 1 after the boot finishes:</p>
<div class="literalblock">
<div class="content">
<pre>./linux/rand_check.out</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is the least reliable setup as there might be other processes that use the given virtual address.</p>
</div>
<div class="sect4">
<h5 id="gdb-step-debug-userland-non-init-without-gdb-wait"><a class="anchor" href="#gdb-step-debug-userland-non-init-without-gdb-wait"></a><a class="link" href="#gdb-step-debug-userland-non-init-without-gdb-wait">3.6.3.1. GDB step debug userland non-init without --gdb-wait</a></h5>
<div class="paragraph">
<p>TODO: if I try <a href="#gdb-step-debug-userland-non-init">GDB step debug userland non-init</a> without <code>--gdb-wait</code> and the <code>break main</code> that we do inside <code>./run-gdb</code> says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Cannot access memory at address 0x10604</pre>
</div>
</div>
<div class="paragraph">
<p>and then GDB never breaks. Tested at ac8663a44a450c3eadafe14031186813f90c21e4 + 1.</p>
</div>
<div class="paragraph">
<p>The exact behaviour seems to depend on the architecture:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>arm</code>: happens always</p>
</li>
<li>
<p><code>x86_64</code>: appears to happen only if you try to connect GDB as fast as possible, before init has been reached.</p>
</li>
<li>
<p><code>aarch64</code>: could not observe the problem</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We have also double checked the address with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch arm readelf -- \
  -s "$(./getvar --arch arm userland_build_dir)/linux/myinsmod.out" | \
  grep main</pre>
</div>
</div>
<div class="paragraph">
<p>and from GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info line main</pre>
</div>
</div>
<div class="paragraph">
<p>and both give:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>000105fc</pre>
</div>
</div>
<div class="paragraph">
<p>which is just 8 bytes before <code>0x10604</code>.</p>
</div>
<div class="paragraph">
<p><code>gdbserver</code> also says <code>0x10604</code>.</p>
</div>
<div class="paragraph">
<p>However, if do a <code>Ctrl-C</code> in GDB, and then a direct:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>b *0x000105fc</pre>
</div>
</div>
<div class="paragraph">
<p>it works. Why?!</p>
</div>
<div class="paragraph">
<p>On GEM5, x86 can also give the <code>Cannot access memory at address</code>, so maybe it is also unreliable on QEMU, and works just by coincidence.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdb-call"><a class="anchor" href="#gdb-call"></a><a class="link" href="#gdb-call">3.7. GDB call</a></h3>
<div class="paragraph">
<p>GDB can call functions as explained at: <a href="https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb" class="bare">https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb</a></p>
</div>
<div class="paragraph">
<p>However this is failing for us:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>some symbols are not visible to <code>call</code> even though <code>b</code> sees them</p>
</li>
<li>
<p>for those that are, <code>call</code> fails with an E14 error</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>E.g.: if we break on <code>__x64_sys_write</code> on <code>count.sh</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&gt;&gt;&gt; call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
&gt;&gt;&gt; b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
&gt;&gt;&gt; call fdget_pos(fd)
No symbol "fdget_pos" in current context.
&gt;&gt;&gt; b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
&gt;&gt;&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>even though <code>fdget_pos</code> is the first thing <code>__x64_sys_write</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);</pre>
</div>
</div>
<div class="paragraph">
<p>I also noticed that I get the same error:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Could not fetch register "orig_rax"; remote failure reply 'E14'</pre>
</div>
</div>
<div class="paragraph">
<p>when trying to use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fin</pre>
</div>
</div>
<div class="paragraph">
<p>on many (all?) functions.</p>
</div>
<div class="paragraph">
<p>See also: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gdb-view-arm-system-registers"><a class="anchor" href="#gdb-view-arm-system-registers"></a><a class="link" href="#gdb-view-arm-system-registers">3.8. GDB view ARM system registers</a></h3>
<div class="paragraph">
<p><code>info all-registers</code> shows some of them.</p>
</div>
<div class="paragraph">
<p>The implementation is described at: <a href="https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044" class="bare">https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gdb-step-debug-multicore-userland"><a class="anchor" href="#gdb-step-debug-multicore-userland"></a><a class="link" href="#gdb-step-debug-multicore-userland">3.9. GDB step debug multicore userland</a></h3>
<div class="paragraph">
<p>For a more minimal baremetal multicore setup, see: <a href="#arm-baremetal-multicore">Section 33.10.3, &#8220;ARM baremetal multicore&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>We can set and get which cores the Linux kernel allows a program to run on with <code>sched_getaffinity</code> and <code>sched_setaffinity</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --eval-after './linux/sched_getaffinity.out'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sched_getaffinity.c">userland/linux/sched_getaffinity.c</a></p>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0</pre>
</div>
</div>
<div class="paragraph">
<p>Which shows us that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>initially:</p>
<div class="ulist">
<ul>
<li>
<p>all 2 cores were enabled as shown by <code>sched_getaffinity = 1 1</code></p>
</li>
<li>
<p>the process was randomly assigned to run on core 1 (the second one) as shown by <code>sched_getcpu = 1</code>. If we run this several times, it will also run on core 0 sometimes.</p>
</li>
</ul>
</div>
</li>
<li>
<p>then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The number of cores is modified as explained at: <a href="#number-of-cores">Section 24.3.1, &#8220;Number of cores&#8221;</a></p>
</div>
<div class="paragraph">
<p><code>taskset</code> from the util-linux package sets the initial core affinity of a program:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'</pre>
</div>
</div>
<div class="paragraph">
<p>output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that the affinity was restricted to the second core from the start.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s do a QEMU observation to justify this example being in the repository with <a href="#gdb-step-debug-userland-non-init">userland breakpoints</a>.</p>
</div>
<div class="paragraph">
<p>We will run our <code>./linux/sched_getaffinity.out</code> infinitely many times, on core 0 and core 1 alternatively:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --cpus 2 \
  --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
  --gdb-wait \
;</pre>
</div>
</div>
<div class="paragraph">
<p>on another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main</pre>
</div>
</div>
<div class="paragraph">
<p>Then, inside GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c</pre>
</div>
</div>
<div class="paragraph">
<p>and we observe that <code>info threads</code> shows the actual correct core on which the process was restricted to run by <code>taskset</code>!</p>
</div>
<div class="paragraph">
<p>We should also try it out with kernel modules: <a href="https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module" class="bare">https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module</a></p>
</div>
<div class="paragraph">
<p>TODO we then tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"</pre>
</div>
</div>
<div class="paragraph">
<p>to switch between two simultaneous live threads with different affinities, it just didn&#8217;t break on our threads:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>b main_thread_0</pre>
</div>
</div>
<div class="paragraph">
<p>Note that secondary cores in gem5 are kind of broken however: <a href="#gem5-gdb-step-debug-secondary-cores">gem5 GDB step debug secondary cores</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/10490756/how-to-use-sched-getaffinity-and-sched-setaffinity-in-linux-from-c/50117787#50117787" class="bare">https://stackoverflow.com/questions/10490756/how-to-use-sched-getaffinity-and-sched-setaffinity-in-linux-from-c/50117787#50117787</a></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/663958/how-to-control-which-core-a-process-runs-on/50210009#50210009" class="bare">https://stackoverflow.com/questions/663958/how-to-control-which-core-a-process-runs-on/50210009#50210009</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/280909/cpu-affinity/54478296#54478296" class="bare">https://stackoverflow.com/questions/280909/cpu-affinity/54478296#54478296</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/73/how-can-i-set-the-processor-affinity-of-a-process-on-linux/441098#441098" class="bare">https://unix.stackexchange.com/questions/73/how-can-i-set-the-processor-affinity-of-a-process-on-linux/441098#441098</a> (summary only)</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/42800801/how-to-use-gdb-to-debug-qemu-with-smp-symmetric-multiple-processors" class="bare">https://stackoverflow.com/questions/42800801/how-to-use-gdb-to-debug-qemu-with-smp-symmetric-multiple-processors</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-gdb-scripts"><a class="anchor" href="#linux-kernel-gdb-scripts"></a><a class="link" href="#linux-kernel-gdb-scripts">3.10. Linux kernel GDB scripts</a></h3>
<div class="paragraph">
<p>We source the Linux kernel GDB scripts by default for <code>lx-symbols</code>, but they also contains some other goodies worth looking into.</p>
</div>
<div class="paragraph">
<p>Those scripts basically parse some in-kernel data structures to offer greater visibility with GDB.</p>
</div>
<div class="paragraph">
<p>All defined commands are prefixed by <code>lx-</code>, so to get a full list just try to tab complete that.</p>
</div>
<div class="paragraph">
<p>There aren&#8217;t as many as I&#8217;d like, and the ones that do exist are pretty self explanatory, but let&#8217;s give a few examples.</p>
</div>
<div class="paragraph">
<p>Show dmesg:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lx-dmesg</pre>
</div>
</div>
<div class="paragraph">
<p>Show the <a href="#kernel-command-line-parameters">Kernel command line parameters</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lx-cmdline</pre>
</div>
</div>
<div class="paragraph">
<p>Dump the device tree to a <code>fdtdump.dtb</code> file in the current directory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lx-fdtdump
pwd</pre>
</div>
</div>
<div class="paragraph">
<p>List inserted kernel modules:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lx-lsmod</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf" class="bare">https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf</a></p>
</li>
<li>
<p><a href="https://wiki.linaro.org/LandingTeams/ST/GDB" class="bare">https://wiki.linaro.org/LandingTeams/ST/GDB</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="lx-ps"><a class="anchor" href="#lx-ps"></a><a class="link" href="#lx-ps">3.10.1. lx-ps</a></h4>
<div class="paragraph">
<p>List all processes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lx-ps</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd</pre>
</div>
</div>
<div class="paragraph">
<p>The second and third fields are obviously PID and process name.</p>
</div>
<div class="paragraph">
<p>The first one is more interesting, and contains the address of the <code>task_struct</code> in memory.</p>
</div>
<div class="paragraph">
<p>This can be confirmed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p ((struct task_struct)*0xffff88000ed08000</pre>
</div>
</div>
<div class="paragraph">
<p>which contains the correct PID for all threads I&#8217;ve tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pid = 1,</pre>
</div>
</div>
<div class="paragraph">
<p>TODO get the PC of the kthreads: <a href="https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel" class="bare">https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel</a> Then we would be able to see where the threads are stopped in the code!</p>
</div>
<div class="paragraph">
<p>On ARM, I tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))-&gt;uregs[ARM_pc]</pre>
</div>
</div>
<div class="paragraph">
<p>but <code>task_pt_regs</code> is a <code>#define</code> and GDB cannot see defines without <code>-ggdb3</code>: <a href="https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb" class="bare">https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb</a> which are apparently not set?</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel" class="bare">https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel</a></p>
</li>
<li>
<p><a href="https://wiki.linaro.org/LandingTeams/ST/GDB" class="bare">https://wiki.linaro.org/LandingTeams/ST/GDB</a></p>
</li>
<li>
<p><a href="https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf" class="bare">https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf</a> presentation: <a href="https://www.youtube.com/watch?v=pqn5hIrz3A8" class="bare">https://www.youtube.com/watch?v=pqn5hIrz3A8</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="config-pid-in-contextidr"><a class="anchor" href="#config-pid-in-contextidr"></a><a class="link" href="#config-pid-in-contextidr">3.10.1.1. CONFIG_PID_IN_CONTEXTIDR</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/54133479/accessing-logical-software-thread-id-in-gem5" class="bare">https://stackoverflow.com/questions/54133479/accessing-logical-software-thread-id-in-gem5</a> on ARM the kernel can store an indication of PID in the CONTEXTIDR_EL1 register, making that much easier to observe from simulators.</p>
</div>
<div class="paragraph">
<p>In particular, gem5 prints that number out by default on <code>ExecAll</code> messages!</p>
</div>
<div class="paragraph">
<p>Let&#8217;s test it out with <a href="#linux-kernel-build-variants">Linux kernel build variants</a> + <a href="#gem5-restore-new-script">gem5 checkpoint restore and run a different script</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --arch aarch64 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --config 'CONFIG_PID_IN_CONTEXTIDR=y'
# Checkpoint run.
./run --arch aarch64 --emulator gem5 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --eval './gem5.sh'
# Trace run.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-readfile 'posix/getpid.out; posix/getpid.out' \
  --gem5-restore 1 \
  --linux-build-id CONFIG_PID_IN_CONTEXTIDR \
  --trace FmtFlag,ExecAll,-ExecSymbol \
;</pre>
</div>
</div>
<div class="paragraph">
<p>The terminal runs both programs which output their PID to stdout:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pid=44
pid=45</pre>
</div>
</div>
<div class="paragraph">
<p>By quickly inspecting the <code>trace.txt</code> file, we immediately notice that the <code>system.cpu: A&lt;n&gt;</code> part of the logs, which used to always be <code>system.cpu: A0</code>, now has a few different values! Nice!</p>
</div>
<div class="paragraph">
<p>We can briefly summarize those values by removing repetitions:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cut -d' ' -f4 "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)" | uniq -c</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  97227 A39
 147476 A38
 222052 A40
      1 terminal
1117724 A40
  27529 A31
  43868 A40
  27487 A31
 138349 A40
  13781 A38
 231246 A40
  25536 A38
  28337 A40
 214799 A38
 963561 A41
  92603 A38
  27511 A31
 224384 A38
 564949 A42
 182360 A38
 729009 A43
   8398 A23
  20200 A10
 636848 A43
 187995 A44
  27529 A31
  70071 A44
  16981 A0
 623806 A44
  16981 A0
 139319 A44
  24487 A0
 174986 A44
  25420 A0
  89611 A44
  16981 A0
 183184 A44
  24728 A0
  89608 A44
  17226 A0
 899075 A44
  24974 A0
 250608 A44
 137700 A43
1497997 A45
 227485 A43
 138147 A38
 482646 A46</pre>
</div>
</div>
<div class="paragraph">
<p>I&#8217;m not smart enough to be able to deduce all of those IDs, but we can at least see that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A44 and A45 are there as expected from stdout!</p>
</li>
<li>
<p>A39 must be the end of the execution of <code>m5 checkpoint</code></p>
</li>
<li>
<p>so we guess that A38 is the shell as it comes next</p>
</li>
<li>
<p>the weird "terminal" line is <code>336969745500: system.terminal: attach terminal 0</code></p>
</li>
<li>
<p>which is the shell PID? I should have printed that as well :-)</p>
</li>
<li>
<p>why are there so many other PIDs? This was supposed to be a silent system without daemons!</p>
</li>
<li>
<p>A0 is presumably the kernel. However we see process switches without going into A0, so I&#8217;m not sure how, it appears to count kernel instructions as part of processes</p>
</li>
<li>
<p>A46 has to be the <code>m5 exit</code> call</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Or if you want to have some real fun, try: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/contextidr_el1.c">baremetal/arch/aarch64/contextidr_el1.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --baremetal baremetal/arch/aarch64/contextidr_el1.c --trace-insts-stdout</pre>
</div>
</div>
<div class="paragraph">
<p>in which we directly set the register ourselves! Output excerpt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  31500: system.cpu: A0 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  32000: system.cpu: A1 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  32500: system.cpu: A1 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  33000: system.cpu: A1 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000002  flags=(IsInteger)
  33500: system.cpu: A1 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  34000: system.cpu: A1 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  34500: system.cpu: A1 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  35000: system.cpu: A1 T0 : @main+40    :   b.le   &lt;main+12&gt;         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  35500: system.cpu: A1 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  36000: system.cpu: A2 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000002  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  36500: system.cpu: A2 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  37000: system.cpu: A2 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000003  flags=(IsInteger)
  37500: system.cpu: A2 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  38000: system.cpu: A2 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  38500: system.cpu: A2 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  39000: system.cpu: A2 T0 : @main+40    :   b.le   &lt;main+12&gt;         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  39500: system.cpu: A2 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  40000: system.cpu: A3 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000003  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)</pre>
</div>
</div>
<div class="paragraph">
<p><a href="#armarm8-fa">ARMv8 architecture reference manual db</a> D13.2.27 "CONTEXTIDR_EL1, Context ID Register (EL1)" documents <code>CONTEXTIDR_EL1</code> as:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Identifies the current Process Identifier.</p>
</div>
<div class="paragraph">
<p>The value of the whole of this register is called the Context ID and is used by:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>The debug logic, for Linked and Unlinked Context ID matching.</p>
</li>
<li>
<p>The trace logic, to identify the current process.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The significance of this register is for debug and trace use only.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Tested on 145769fc387dc5ee63ec82e55e6b131d9c968538 + 1.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="debug-the-gdb-remote-protocol"><a class="anchor" href="#debug-the-gdb-remote-protocol"></a><a class="link" href="#debug-the-gdb-remote-protocol">3.11. Debug the GDB remote protocol</a></h3>
<div class="paragraph">
<p>For when it breaks again, or you want to add a new feature!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug
./run-gdb --before '-ex "set remotetimeout 99999" -ex "set debug remote 1"' start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets" class="bare">https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets</a></p>
</div>
<div class="sect3">
<h4 id="remote-g-packet"><a class="anchor" href="#remote-g-packet"></a><a class="link" href="#remote-g-packet">3.11.1. Remote 'g' packet reply is too long</a></h4>
<div class="paragraph">
<p>This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.</p>
</div>
<div class="paragraph">
<p>This can happen for the following reasons:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>you set the architecture of the client wrong, often 32 vs 64 bit as mentioned at: <a href="https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long" class="bare">https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long</a></p>
</li>
<li>
<p>there is a bug in the GDB server and the XML description does not match the number of registers actually sent</p>
</li>
<li>
<p>the GDB server does not send XML target descriptions and your GDB expects a different number of registers by default. E.g., gem5 d4b3e064adeeace3c3e7d106801f95c14637c12f does not send the XML files</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The XML target description format is described a bit further at: <a href="https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044" class="bare">https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044</a></p>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="kgdb"><a class="anchor" href="#kgdb"></a><a class="link" href="#kgdb">4. KGDB</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>KGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support.</p>
</div>
<div class="paragraph">
<p>It is useless with QEMU since we already have full system visibility with <code>-gdb</code>. So the goal of this setup is just to prepare you for what to expect when you will be in the treches of real hardware.</p>
</div>
<div class="paragraph">
<p>KGDB is cheaper than JTAG (free) and easier to setup (all you need is serial), but with less visibility as it depends on the kernel working, so e.g.: dies on panic, does not see boot sequence.</p>
</div>
<div class="paragraph">
<p>First run the kernel with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kgdb</pre>
</div>
</div>
<div class="paragraph">
<p>this passes the following options on the kernel CLI:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>kgdbwait kgdboc=ttyS1,115200</pre>
</div>
</div>
<div class="paragraph">
<p><code>kgdbwait</code> tells the kernel to wait for KGDB to connect.</p>
</div>
<div class="paragraph">
<p>So the kernel sets things up enough for KGDB to start working, and then boot pauses waiting for connection:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    4.866050] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
&lt;6&gt;[    4.893205] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
&lt;6&gt;[    4.916271] 00:06: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
&lt;6&gt;[    4.987771] KGDB: Registered I/O driver kgdboc
&lt;2&gt;[    4.996053] KGDB: Waiting for connection from remote gdb...

Entering kdb (current=0x(____ptrval____), pid 1) on processor 0 due to Keyboard Entry
[0]kdb&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>KGDB expects the connection at <code>ttyS1</code>, our second serial port after <code>ttyS0</code> which contains the terminal.</p>
</div>
<div class="paragraph">
<p>The last line is the KDB prompt, and is covered at: <a href="#kdb">Section 4.3, &#8220;KDB&#8221;</a>. Typing now shows nothing because that prompt is expecting input from <code>ttyS1</code>.</p>
</div>
<div class="paragraph">
<p>Instead, we connect to the serial port <code>ttyS1</code> with GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --kgdb --no-continue</pre>
</div>
</div>
<div class="paragraph">
<p>Once GDB connects, it is left inside the function <code>kgdb_breakpoint</code>.</p>
</div>
<div class="paragraph">
<p>So now we can set breakpoints and continue as usual.</p>
</div>
<div class="paragraph">
<p>For example, in GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>continue</pre>
</div>
</div>
<div class="paragraph">
<p>Then in QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./count.sh &amp;
./kgdb.sh</pre>
</div>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/kgdb.sh">rootfs_overlay/lkmc/kgdb.sh</a> pauses the kernel for KGDB, and gives control back to GDB.</p>
</div>
<div class="paragraph">
<p>And now in GDB we do the usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>break __x64_sys_write
continue
continue
continue
continue</pre>
</div>
</div>
<div class="paragraph">
<p>And now you can count from KGDB!</p>
</div>
<div class="paragraph">
<p>If you do: <code>break __x64_sys_write</code> immediately after <code>./run-gdb --kgdb</code>, it fails with <code>KGDB: BP remove failed: &lt;address&gt;</code>. I think this is because it would break too early on the boot sequence, and KGDB is not yet ready.</p>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.9/Documentation/DocBook/kgdb.tmpl" class="bare">https://github.com/torvalds/linux/blob/v4.9/Documentation/DocBook/kgdb.tmpl</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/22004616/qemu-kernel-debugging-with-kgdb/44197715#44197715" class="bare">https://stackoverflow.com/questions/22004616/qemu-kernel-debugging-with-kgdb/44197715#44197715</a></p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="kgdb-arm"><a class="anchor" href="#kgdb-arm"></a><a class="link" href="#kgdb-arm">4.1. KGDB ARM</a></h3>
<div class="paragraph">
<p>TODO: we would need a second serial for KGDB to work, but it is not currently supported on <code>arm</code> and <code>aarch64</code> with <code>-M virt</code> that we use: <a href="https://unix.stackexchange.com/questions/479085/can-qemu-m-virt-on-arm-aarch64-have-multiple-serial-ttys-like-such-as-pl011-t/479340#479340" class="bare">https://unix.stackexchange.com/questions/479085/can-qemu-m-virt-on-arm-aarch64-have-multiple-serial-ttys-like-such-as-pl011-t/479340#479340</a></p>
</div>
<div class="paragraph">
<p>One possible workaround for this would be to use <a href="#kdb-arm">KDB ARM</a>.</p>
</div>
<div class="paragraph">
<p>Main more generic question: <a href="https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm" class="bare">https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm</a></p>
</div>
</div>
<div class="sect2">
<h3 id="kgdb-kernel-modules"><a class="anchor" href="#kgdb-kernel-modules"></a><a class="link" href="#kgdb-kernel-modules">4.2. KGDB kernel modules</a></h3>
<div class="paragraph">
<p>Just works as you would expect:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod timer.ko
./kgdb.sh</pre>
</div>
</div>
<div class="paragraph">
<p>In GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>break lkmc_timer_callback
continue
continue
continue</pre>
</div>
</div>
<div class="paragraph">
<p>and you now control the count.</p>
</div>
</div>
<div class="sect2">
<h3 id="kdb"><a class="anchor" href="#kdb"></a><a class="link" href="#kdb">4.3. KDB</a></h3>
<div class="paragraph">
<p>KDB is a way to use KDB directly in your main console, without GDB.</p>
</div>
<div class="paragraph">
<p>Advantage over KGDB: you can do everything in one serial. This can actually be important if you only have one serial for both shell and .</p>
</div>
<div class="paragraph">
<p>Disadvantage: not as much functionality as GDB, especially when you use Python scripts. Notably, TODO confirm you can&#8217;t see the the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source, or if the kernel had a crazy mechanism to embed it).</p>
</div>
<div class="paragraph">
<p>Run QEMU as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kdb</pre>
</div>
</div>
<div class="paragraph">
<p>This passes <code>kgdboc=ttyS0</code> to the Linux CLI, therefore using our main console. Then QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[0]kdb&gt; go</pre>
</div>
</div>
<div class="paragraph">
<p>And now the <code>kdb&gt;</code> prompt is responsive because it is listening to the main console.</p>
</div>
<div class="paragraph">
<p>After boot finishes, run the usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./count.sh &amp;
./kgdb.sh</pre>
</div>
</div>
<div class="paragraph">
<p>And you are back in KDB. Now you can count with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[0]kdb&gt; bp __x64_sys_write
[0]kdb&gt; go
[0]kdb&gt; go
[0]kdb&gt; go
[0]kdb&gt; go</pre>
</div>
</div>
<div class="paragraph">
<p>And you will break whenever <code>__x64_sys_write</code> is hit.</p>
</div>
<div class="paragraph">
<p>You can get see further commands with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[0]kdb&gt; help</pre>
</div>
</div>
<div class="paragraph">
<p>The other KDB commands allow you to step instructions, view memory, registers and some higher level kernel runtime data similar to the superior GDB Python scripts.</p>
</div>
<div class="sect3">
<h4 id="kdb-graphic"><a class="anchor" href="#kdb-graphic"></a><a class="link" href="#kdb-graphic">4.3.1. KDB graphic</a></h4>
<div class="paragraph">
<p>You can also use KDB directly from the <a href="#graphics">graphic</a> window with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --graphic --kdb</pre>
</div>
</div>
<div class="paragraph">
<p>This setup could be used to debug the kernel on machines without serial, such as modern desktops.</p>
</div>
<div class="paragraph">
<p>This works because <code>--graphics</code> adds <code>kbd</code> (which stands for <code>KeyBoarD</code>!) to <code>kgdboc</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="kdb-arm"><a class="anchor" href="#kdb-arm"></a><a class="link" href="#kdb-arm">4.3.2. KDB ARM</a></h4>
<div class="paragraph">
<p>TODO neither <code>arm</code> and <code>aarch64</code> are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.</p>
</div>
<div class="paragraph">
<p><code>arm</code> seems to place and hit the breakpoint correctly, but no matter how many <code>go</code> commands I do, the <code>count.sh</code> stdout simply does not show.</p>
</div>
<div class="paragraph">
<p><code>aarch64</code> seems to place the breakpoint correctly, but after the first <code>go</code> the kernel oopses with warning:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>WARNING: CPU: 0 PID: 46 at /root/linux-kernel-module-cheat/submodules/linux/kernel/smp.c:416 smp_call_function_many+0xdc/0x358</pre>
</div>
</div>
<div class="paragraph">
<p>and stack trace:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>smp_call_function_many+0xdc/0x358
kick_all_cpus_sync+0x30/0x38
kgdb_flush_swbreak_addr+0x3c/0x48
dbg_deactivate_sw_breakpoints+0x7c/0xb8
kgdb_cpu_enter+0x284/0x6a8
kgdb_handle_exception+0x138/0x240
kgdb_brk_fn+0x2c/0x40
brk_handler+0x7c/0xc8
do_debug_exception+0xa4/0x1c0
el1_dbg+0x18/0x78
__arm64_sys_write+0x0/0x30
el0_svc_handler+0x74/0x90
el0_svc+0x8/0xc</pre>
</div>
</div>
<div class="paragraph">
<p>My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="gdbserver"><a class="anchor" href="#gdbserver"></a><a class="link" href="#gdbserver">5. gdbserver</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Step debug userland processes to understand how they are talking to the kernel.</p>
</div>
<div class="paragraph">
<p>First build <code>gdbserver</code> into the root filesystem:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_GDB=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Then on guest, to debug <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/rand_check.c">userland/linux/rand_check.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gdbserver.sh ./c/command_line_arguments.out asdf qwer</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/gdbserver.sh">rootfs_overlay/lkmc/gdbserver.sh</a>.</p>
</div>
<div class="paragraph">
<p>And on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --gdbserver --userland userland/c/command_line_arguments.c main</pre>
</div>
</div>
<div class="paragraph">
<p>or alternatively with the path to the executable itself:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdbserver --userland "$(./getvar userland_build_dir)/c/command_line_arguments.out"</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-arm-mips-elf-with-qemu-toolchain/16214#16214" class="bare">https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-arm-mips-elf-with-qemu-toolchain/16214#16214</a></p>
</div>
<div class="sect2">
<h3 id="gdbserver-busybox"><a class="anchor" href="#gdbserver-busybox"></a><a class="link" href="#gdbserver-busybox">5.1. gdbserver BusyBox</a></h3>
<div class="paragraph">
<p>Analogous to <a href="#gdb-step-debug-userland-processes">GDB step debug userland processes</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gdbserver.sh ls</pre>
</div>
</div>
<div class="paragraph">
<p>on host you need:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --gdbserver --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gdbserver-libc"><a class="anchor" href="#gdbserver-libc"></a><a class="link" href="#gdbserver-libc">5.2. gdbserver libc</a></h3>
<div class="paragraph">
<p>Our setup gives you the rare opportunity to step debug libc and other system libraries.</p>
</div>
<div class="paragraph">
<p>For example in the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gdbserver.sh ./posix/count.out</pre>
</div>
</div>
<div class="paragraph">
<p>Then on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --gdbserver --userland userland/posix/count.c main</pre>
</div>
</div>
<div class="paragraph">
<p>and inside GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>break sleep
continue</pre>
</div>
</div>
<div class="paragraph">
<p>And you are now left inside the <code>sleep</code> function of our default libc implementation uclibc <a href="https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/unistd/sleep.c?h=v1.0.30#n91"><code>libc/unistd/sleep.c</code></a>!</p>
</div>
<div class="paragraph">
<p>You can also step into the <code>sleep</code> call:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>step</pre>
</div>
</div>
<div class="paragraph">
<p>This is made possible by the GDB command that we use by default:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>set sysroot ${common_buildroot_build_dir}/staging</pre>
</div>
</div>
<div class="paragraph">
<p>which automatically finds unstripped shared libraries on the host for us.</p>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113" class="bare">https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gdbserver-dynamic-loader"><a class="anchor" href="#gdbserver-dynamic-loader"></a><a class="link" href="#gdbserver-dynamic-loader">5.3. gdbserver dynamic loader</a></h3>
<div class="paragraph">
<p>TODO: try to step debug the dynamic loader. Would be even easier if <code>starti</code> is available: <a href="https://stackoverflow.com/questions/10483544/stopping-at-the-first-machine-code-instruction-in-gdb" class="bare">https://stackoverflow.com/questions/10483544/stopping-at-the-first-machine-code-instruction-in-gdb</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/20114565/gdb-step-into-dynamic-linkerld-so-code" class="bare">https://stackoverflow.com/questions/20114565/gdb-step-into-dynamic-linkerld-so-code</a></p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="cpu-architecture"><a class="anchor" href="#cpu-architecture"></a><a class="link" href="#cpu-architecture">6. CPU architecture</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware.</p>
</div>
<div class="paragraph">
<p>To use <code>arm</code> instead of x86 for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch arm
./run --arch arm</pre>
</div>
</div>
<div class="paragraph">
<p>Debug:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --gdb-wait
# On another terminal.
./run-gdb --arch arm</pre>
</div>
</div>
<div class="paragraph">
<p>We also have one letter shorthand names for the architectures and <code>--arch</code> option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># aarch64
./run -a A
# arm
./run -a a
# x86_64
./run -a x</pre>
</div>
</div>
<div class="paragraph">
<p>Known quirks of the supported architectures are documented in this section.</p>
</div>
<div class="sect2">
<h3 id="x86-64"><a class="anchor" href="#x86-64"></a><a class="link" href="#x86-64">6.1. x86_64</a></h3>
<div class="sect3">
<h4 id="ring0"><a class="anchor" href="#ring0"></a><a class="link" href="#ring0">6.1.1. ring0</a></h4>
<div class="paragraph">
<p>This example illustrates how reading from the x86 control registers with <code>mov crX, rax</code> can only be done from kernel land on ring0.</p>
</div>
<div class="paragraph">
<p>From kernel land:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod ring0.ko</pre>
</div>
</div>
<div class="paragraph">
<p>works and output the registers, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cr0 = 0xFFFF880080050033
cr2 = 0xFFFFFFFF006A0008
cr3 = 0xFFFFF0DCDC000</pre>
</div>
</div>
<div class="paragraph">
<p>However if we try to do it from userland:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./ring0.out</pre>
</div>
</div>
<div class="paragraph">
<p>stdout gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Segmentation fault</pre>
</div>
</div>
<div class="paragraph">
<p>and dmesg outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>traps: ring0.out[55] general protection ip:40054c sp:7fffffffec20 error:0 in ring0.out[400000+1000]</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/ring0.c">kernel_modules/ring0.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/ring0.h">lkmc/ring0.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/ring0.c">userland/arch/x86_64/ring0.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In both cases, we attempt to run the exact same code which is shared on the <code>ring0.h</code> header file.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306" class="bare">https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439" class="bare">https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm"><a class="anchor" href="#arm"></a><a class="link" href="#arm">6.2. arm</a></h3>
<div class="sect3">
<h4 id="run-arm-executable-in-aarch64"><a class="anchor" href="#run-arm-executable-in-aarch64"></a><a class="link" href="#run-arm-executable-in-aarch64">6.2.1. Run arm executable in aarch64</a></h4>
<div class="paragraph">
<p>TODO Can you run arm executables in the aarch64 guest? <a href="https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709" class="bare">https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709</a></p>
</div>
<div class="paragraph">
<p>I&#8217;ve tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch aarch64 gcc -- -static ~/test/hello_world.c -o "$(./getvar p9_dir)/a.out"
./run --arch aarch64 --eval-after '/mnt/9p/data/a.out'</pre>
</div>
</div>
<div class="paragraph">
<p>but it fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>a.out: line 1: syntax error: unexpected word (expecting ")")</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="mips"><a class="anchor" href="#mips"></a><a class="link" href="#mips">6.3. MIPS</a></h3>
<div class="paragraph">
<p>We used to "support" it until f8c0502bb2680f2dbe7c1f3d7958f60265347005 (it booted) but dropped since one was testing it often.</p>
</div>
<div class="paragraph">
<p>If you want to revive and maintain it, send a pull request.</p>
</div>
</div>
<div class="sect2">
<h3 id="other-architectures"><a class="anchor" href="#other-architectures"></a><a class="link" href="#other-architectures">6.4. Other architectures</a></h3>
<div class="paragraph">
<p>It should not be too hard to port this repository to any architecture that Buildroot supports. Pull requests are welcome.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="init"><a class="anchor" href="#init"></a><a class="link" href="#init">7. init</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>When the Linux kernel finishes booting, it runs an executable as the first and only userland process. This executable is called the <code>init</code> program.</p>
</div>
<div class="paragraph">
<p>The init process is then responsible for setting up the entire userland (or destroying everything when you want to have fun).</p>
</div>
<div class="paragraph">
<p>This typically means reading some configuration files (e.g. <code>/etc/initrc</code>) and forking a bunch of userland executables based on those files, including the very interactive shell that we end up on.</p>
</div>
<div class="paragraph">
<p>systemd provides a "popular" init implementation for desktop distros as of 2017.</p>
</div>
<div class="paragraph">
<p>BusyBox provides its own minimalistic init implementation which Buildroot, and therefore this repo, uses by default.</p>
</div>
<div class="paragraph">
<p>The <code>init</code> program can be either an executable shell text file, or a compiled ELF file. It becomes easy to accept this once you see that the <code>exec</code> system call handles both cases equally: <a href="https://unix.stackexchange.com/questions/174062/can-the-init-process-be-a-shell-script-in-linux/395375#395375" class="bare">https://unix.stackexchange.com/questions/174062/can-the-init-process-be-a-shell-script-in-linux/395375#395375</a></p>
</div>
<div class="paragraph">
<p>The <code>init</code> executable is searched for in a list of paths in the root filesystem, including <code>/init</code>, <code>/sbin/init</code> and a few others. For more details see: <a href="#path-to-init">Section 7.3, &#8220;Path to init&#8221;</a></p>
</div>
<div class="sect2">
<h3 id="replace-init"><a class="anchor" href="#replace-init"></a><a class="link" href="#replace-init">7.1. Replace init</a></h3>
<div class="paragraph">
<p>To have more control over the system, you can replace BusyBox&#8217;s init with your own.</p>
</div>
<div class="paragraph">
<p>The most direct way to replace <code>init</code> with our own is to just use the <code>init=</code> <a href="#kernel-command-line-parameters">command line parameter</a> directly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/lkmc/count.sh'</pre>
</div>
</div>
<div class="paragraph">
<p>This just counts every second forever and does not give you a shell.</p>
</div>
<div class="paragraph">
<p>This method is not very flexible however, as it is hard to reliably pass multiple commands and command line arguments to the init with it, as explained at: <a href="#init-environment">Section 7.4, &#8220;Init environment&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>For this reason, we have created a more robust helper method with the <code>--eval</code> option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'echo "asdf qwer";insmod hello.ko;./linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>It is basically a shortcut for:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/lkmc/eval_base64.sh - lkmc_eval="insmod hello.ko;./linux/poweroff.out"'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/eval_base64.sh">rootfs_overlay/lkmc/eval_base64.sh</a>.</p>
</div>
<div class="paragraph">
<p>This allows quoting and newlines by base64 encoding on host, and decoding on guest, see: <a href="#kernel-command-line-parameters-escaping">Section 17.3.1, &#8220;Kernel command line parameters escaping&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>It also automatically chooses between <code>init=</code> and <code>rcinit=</code> for you, see: <a href="#path-to-init">Section 7.3, &#8220;Path to init&#8221;</a></p>
</div>
<div class="paragraph">
<p><code>--eval</code> replaces BusyBox' init completely, which makes things more minimal, but also has has the following consequences:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/etc/fstab</code> mounts are not done, notably <code>/proc</code> and <code>/sys</code>, test it out with:</p>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'echo asdf;ls /proc;ls /sys;echo qwer'</pre>
</div>
</div>
</li>
<li>
<p>no shell is launched at the end of boot for you to interact with the system. You could explicitly add a <code>sh</code> at the end of your commands however:</p>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'echo hello;sh'</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The best way to overcome those limitations is to use: <a href="#init-busybox">Section 7.2, &#8220;Run command at the end of BusyBox init&#8221;</a></p>
</div>
<div class="paragraph">
<p>If the script is large, you can add it to a gitignored file and pass that to <code>--eval</code> as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo '
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' &gt; data/gitignore.sh
./run --eval "$(cat data/gitignore.sh)"</pre>
</div>
</div>
<div class="paragraph">
<p>or add it to a file to the root filesystem guest and rebuild:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo '#!/bin/sh
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' &gt; rootfs_overlay/lkmc/gitignore.sh
chmod +x rootfs_overlay/lkmc/gitignore.sh
./build-buildroot
./run --kernel-cli 'init=/lkmc/gitignore.sh'</pre>
</div>
</div>
<div class="paragraph">
<p>Remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>run forever in a loop or long sleep</p>
</li>
<li>
<p><code>poweroff</code> the machine</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="poweroff-out"><a class="anchor" href="#poweroff-out"></a><a class="link" href="#poweroff-out">7.1.1. poweroff.out</a></h4>
<div class="paragraph">
<p>Just using BusyBox' <code>poweroff</code> at the end of the <code>init</code> does not work and the kernel panics:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval poweroff</pre>
</div>
</div>
<div class="paragraph">
<p>because BusyBox' <code>poweroff</code> tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.</p>
</div>
<div class="paragraph">
<p>But this fails when we are <code>init</code> itself!</p>
</div>
<div class="paragraph">
<p>BusyBox' <code>poweroff</code> works more brutally and effectively if you add <code>-f</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'poweroff -f'</pre>
</div>
</div>
<div class="paragraph">
<p>but why not just use our minimal <code>./linux/poweroff.out</code> and be done with it?</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval './linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/poweroff.c">userland/linux/poweroff.c</a></p>
</div>
<div class="paragraph">
<p>This also illustrates how to shutdown the computer from C: <a href="https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system" class="bare">https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system</a></p>
</div>
</div>
<div class="sect3">
<h4 id="sleep-forever-out"><a class="anchor" href="#sleep-forever-out"></a><a class="link" href="#sleep-forever-out">7.1.2. sleep_forever.out</a></h4>
<div class="paragraph">
<p>I dare you to guess what this does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval './posix/sleep_forever.out'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/sleep_forever.c">userland/posix/sleep_forever.c</a></p>
</div>
<div class="paragraph">
<p>This executable is a convenient simple init that does not panic and sleeps instead.</p>
</div>
</div>
<div class="sect3">
<h4 id="time-boot-out"><a class="anchor" href="#time-boot-out"></a><a class="link" href="#time-boot-out">7.1.3. time_boot.out</a></h4>
<div class="paragraph">
<p>Get a reasonable answer to "how long does boot take in guest time?":</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './linux/time_boot.c'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/time_boot.c">userland/linux/time_boot.c</a></p>
</div>
<div class="paragraph">
<p>That executable writes to <code>dmesg</code> directly through <code>/dev/kmsg</code> a message of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    2.188242] /path/to/linux-kernel-module-cheat/userland/linux/time_boot.c</pre>
</div>
</div>
<div class="paragraph">
<p>which tells us that boot took <code>2.188242</code> seconds based on the dmesg timestamp.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014" class="bare">https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="init-busybox"><a class="anchor" href="#init-busybox"></a><a class="link" href="#init-busybox">7.2. Run command at the end of BusyBox init</a></h3>
<div class="paragraph">
<p>Use the <code>--eval-after</code> option is for you rely on something that BusyBox' init set up for you like <code>/etc/fstab</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'echo asdf;ls /proc;ls /sys;echo qwer'</pre>
</div>
</div>
<div class="paragraph">
<p>After the commands run, you are left on an interactive shell.</p>
</div>
<div class="paragraph">
<p>The above command is basically equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli-after-dash 'lkmc_eval="insmod hello.ko;./linux/poweroff.out;"'</pre>
</div>
</div>
<div class="paragraph">
<p>where the <code>lkmc_eval</code> option gets evaled by our default <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/init.d/S98">rootfs_overlay/etc/init.d/S98</a> startup script.</p>
</div>
<div class="paragraph">
<p>Except that <code>--eval-after</code> is smarter and uses <code>base64</code> encoding.</p>
</div>
<div class="paragraph">
<p>Alternatively, you can also add the comamdns to run to a new <code>init.d</code> entry to run at the end o the BusyBox init:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build-buildroot
./run</pre>
</div>
</div>
<div class="paragraph">
<p>and they will be run automatically before the login prompt.</p>
</div>
<div class="paragraph">
<p>Scripts under <code>/etc/init.d</code> are run by <code>/etc/init.d/rcS</code>, which gets called by the line <code>::sysinit:/etc/init.d/rcS</code> in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/inittab"><code>/etc/inittab</code></a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="path-to-init"><a class="anchor" href="#path-to-init"></a><a class="link" href="#path-to-init">7.3. Path to init</a></h3>
<div class="paragraph">
<p>The init is selected at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>initrd or initramfs system: <code>/init</code>, a custom one can be set with the <code>rdinit=</code> <a href="#kernel-command-line-parameters">kernel command line parameter</a></p>
</li>
<li>
<p>otherwise: default is <code>/sbin/init</code>, followed by some other paths, a custom one can be set with <code>init=</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>More details: <a href="https://unix.stackexchange.com/questions/30414/what-can-make-passing-init-path-to-program-to-the-kernel-not-start-program-as-i/430614#430614" class="bare">https://unix.stackexchange.com/questions/30414/what-can-make-passing-init-path-to-program-to-the-kernel-not-start-program-as-i/430614#430614</a></p>
</div>
<div class="paragraph">
<p>The final init that actually got selected is shown on Linux v5.9.2 a line of type:</p>
</div>
<div class="listingblock">
<div class="content">
<pre class="highlight"><code>&lt;6&gt;[    0.309984] Run /sbin/init as init process</code></pre>
</div>
</div>
<div class="paragraph">
<p>at the very end of the boot logs.</p>
</div>
</div>
<div class="sect2">
<h3 id="init-environment"><a class="anchor" href="#init-environment"></a><a class="link" href="#init-environment">7.4. Init environment</a></h3>
<div class="paragraph">
<p>Documented at <a href="https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html" class="bare">https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html</a>:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The kernel parses parameters from the kernel command line up to "-"; if it doesn&#8217;t recognize a parameter and it doesn&#8217;t contain a '.', the parameter gets passed to init: parameters with '=' go into init&#8217;s environment, others are passed as command line arguments to init. Everything after "-" is passed as an argument to init.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>And you can try it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash 'asdf=qwer zxcv'</pre>
</div>
</div>
<div class="paragraph">
<p>From the <a href="#dry-run">generated QEMU command</a>, we see that the kernel CLI at LKMC 69f5745d3df11d5c741551009df86ea6c61a09cf now contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>init=/lkmc/linux/init_env_poweroff.out console=ttyS0 - lkmc_home=/lkmc asdf=qwer zxcv</pre>
</div>
</div>
<div class="paragraph">
<p>and the init program outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>args:
/lkmc/linux/init_env_poweroff.out
-
zxcv

env:
HOME=/
TERM=linux
lkmc_home=/lkmc
asdf=qwer</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/init_env_poweroff.c">userland/linux/init_env_poweroff.c</a>.</p>
</div>
<div class="paragraph">
<p>As of the Linux kernel v5.7 (possibly earlier, I&#8217;ve skipped a few releases), boot also shows the init arguments and environment very clearly, which is a great addition:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    0.309984] Run /sbin/init as init process
&lt;7&gt;[    0.309991]   with arguments:
&lt;7&gt;[    0.309997]     /sbin/init
&lt;7&gt;[    0.310004]     nokaslr
&lt;7&gt;[    0.310010]     -
&lt;7&gt;[    0.310016]   with environment:
&lt;7&gt;[    0.310022]     HOME=/
&lt;7&gt;[    0.310028]     TERM=linux
&lt;7&gt;[    0.310035]     earlyprintk=pl011,0x1c090000
&lt;7&gt;[    0.310041]     lkmc_home=/lkmc</pre>
</div>
</div>
<div class="sect3">
<h4 id="init-arguments"><a class="anchor" href="#init-arguments"></a><a class="link" href="#init-arguments">7.4.1. init arguments</a></h4>
<div class="paragraph">
<p>The annoying dash <code>-</code> gets passed as a parameter to <code>init</code>, which makes it impossible to use this method for most non custom executables.</p>
</div>
<div class="paragraph">
<p>Arguments with dots that come after <code>-</code> are still treated specially (of the form <code>subsystem.somevalue</code>) and disappear, from args, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash '/lkmc/linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>args
/lkmc/linux/init_env_poweroff.out
-
ab</pre>
</div>
</div>
<div class="paragraph">
<p>so see how <code>a.b</code> is gone.</p>
</div>
<div class="paragraph">
<p>The simple workaround is to just create a shell script that does it, e.g. as we&#8217;ve done at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/gem5_exit.sh">rootfs_overlay/lkmc/gem5_exit.sh</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="init-environment-env"><a class="anchor" href="#init-environment-env"></a><a class="link" href="#init-environment-env">7.4.2. init environment env</a></h4>
<div class="paragraph">
<p>Wait, where do <code>HOME</code> and <code>TERM</code> come from? (greps the kernel). Ah, OK, the kernel sets those by default: <a href="https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173" class="bare">https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>const char *envp_init[MAX_INIT_ENVS+2] = { "HOME=/", "TERM=linux", NULL, };</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="busybox-shell-init-environment"><a class="anchor" href="#busybox-shell-init-environment"></a><a class="link" href="#busybox-shell-init-environment">7.4.3. BusyBox shell init environment</a></h4>
<div class="paragraph">
<p>On top of the Linux kernel, the BusyBox <code>/bin/sh</code> shell will also define other variables.</p>
</div>
<div class="paragraph">
<p>We can explore the shenanigans that the shell adds on top of the Linux kernel with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/bin/sh'</pre>
</div>
</div>
<div class="paragraph">
<p>From there we observe that:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>env</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>SHLVL=1
HOME=/
TERM=linux
PWD=/</pre>
</div>
</div>
<div class="paragraph">
<p>therefore adding <code>SHLVL</code> and <code>PWD</code> to the default kernel exported variables.</p>
</div>
<div class="paragraph">
<p>Furthermore, to increase confusion, if you list all non-exported shell variables <a href="https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values" class="bare">https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>set</pre>
</div>
</div>
<div class="paragraph">
<p>then it shows more variables, notably:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>PATH='/sbin:/usr/sbin:/bin:/usr/bin'</pre>
</div>
</div>
<div class="sect4">
<h5 id="busybox-shell-initrc-files"><a class="anchor" href="#busybox-shell-initrc-files"></a><a class="link" href="#busybox-shell-initrc-files">7.4.3.1. BusyBox shell initrc files</a></h5>
<div class="paragraph">
<p>Login shells source some default files, notably:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/etc/profile
$HOME/.profile</pre>
</div>
</div>
<div class="paragraph">
<p>In our case, <code>HOME</code> is set to <code>/</code> presumably by <code>init</code> at: <a href="https://git.busybox.net/busybox/tree/init/init.c?id=5059653882dbd86e3bbf48389f9f81b0fac8cd0a#n1114" class="bare">https://git.busybox.net/busybox/tree/init/init.c?id=5059653882dbd86e3bbf48389f9f81b0fac8cd0a#n1114</a></p>
</div>
<div class="paragraph">
<p>We provide <code>/.profile</code> from <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/.profile">rootfs_overlay/.profile</a>, and use the default BusyBox <code>/etc/profile</code>.</p>
</div>
<div class="paragraph">
<p>The shell knows that it is a login shell if the first character of <code>argv[0]</code> is <code>-</code>, see also: <a href="https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142" class="bare">https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142</a></p>
</div>
<div class="paragraph">
<p>When we use just <code>init=/bin/sh</code>, the Linux kernel sets <code>argv[0]</code> to <code>/bin/sh</code>, which does not start with <code>-</code>.</p>
</div>
<div class="paragraph">
<p>However, if you use <code>::respawn:-/bin/sh</code> on inttab described at <a href="#tty">TTY</a>, BusyBox' init sets <code>argv[0][0]</code> to <code>-</code>, and so does <code>getty</code>. This can be observed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/$$/cmdline</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>$$</code> is the PID of the shell itself: <a href="https://stackoverflow.com/questions/21063765/get-pid-in-shell-bash" class="bare">https://stackoverflow.com/questions/21063765/get-pid-in-shell-bash</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://unix.stackexchange.com/questions/176027/ash-profile-configuration-file" class="bare">https://unix.stackexchange.com/questions/176027/ash-profile-configuration-file</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="initrd"><a class="anchor" href="#initrd"></a><a class="link" href="#initrd">8. initrd</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The kernel can boot from an CPIO file, which is a directory serialization format much like tar: <a href="https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference" class="bare">https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference</a></p>
</div>
<div class="paragraph">
<p>The bootloader, which for us is provided by QEMU itself, is then configured to put that CPIO into memory, and tell the kernel that it is there.</p>
</div>
<div class="paragraph">
<p>This is very similar to the kernel image itself, which already gets put into memory by the QEMU <code>-kernel</code> option.</p>
</div>
<div class="paragraph">
<p>With this setup, you don&#8217;t even need to give a root filesystem to the kernel: it just does everything in memory in a ramfs.</p>
</div>
<div class="paragraph">
<p>To enable initrd instead of the default ext2 disk image, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --initrd
./run --initrd</pre>
</div>
</div>
<div class="paragraph">
<p>By looking at the QEMU run command generated, you can see that we didn&#8217;t give the <code>-drive</code> option at all:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar run_dir)/run.sh"</pre>
</div>
</div>
<div class="paragraph">
<p>Instead, we used the QEMU <code>-initrd</code> option to point to the <code>.cpio</code> filesystem that Buildroot generated for us.</p>
</div>
<div class="paragraph">
<p>Try removing that <code>-initrd</code> option to watch the kernel panic without rootfs at the end of boot.</p>
</div>
<div class="paragraph">
<p>When using <code>.cpio</code>, there can be no <a href="#disk-persistency">filesystem persistency</a> across boots, since all file operations happen in memory in a tmpfs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>date &gt;f
poweroff
cat f
# can't open 'f': No such file or directory</pre>
</div>
</div>
<div class="paragraph">
<p>which can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.</p>
</div>
<div class="paragraph">
<p>Not however that we already disable disk persistency by default on ext2 filesystems even without <code>--initrd</code>: <a href="#disk-persistency">Section 23.3, &#8220;Disk persistency&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>One downside of this method is that it has to put the entire filesystem into memory, and could lead to a panic:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>end Kernel panic - not syncing: Out of memory and no killable processes...</pre>
</div>
</div>
<div class="paragraph">
<p>This can be solved by increasing the memory as explained at <a href="#memory-size">Memory size</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --initrd --memory 256M</pre>
</div>
</div>
<div class="paragraph">
<p>The main ingredients to get initrd working are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>BR2_TARGET_ROOTFS_CPIO=y</code>: make Buildroot generate <code>images/rootfs.cpio</code> in addition to the other images.</p>
<div class="paragraph">
<p>It is also possible to compress that image with other options.</p>
</div>
</li>
<li>
<p><code>qemu -initrd</code>: make QEMU put the image into memory and tell the kernel about it.</p>
</li>
<li>
<p><code>CONFIG_BLK_DEV_INITRD=y</code>: Compile the kernel with initrd support, see also: <a href="https://unix.stackexchange.com/questions/67462/linux-kernel-is-not-finding-the-initrd-correctly/424496#424496" class="bare">https://unix.stackexchange.com/questions/67462/linux-kernel-is-not-finding-the-initrd-correctly/424496#424496</a></p>
<div class="paragraph">
<p>Buildroot forces that option when <code>BR2_TARGET_ROOTFS_CPIO=y</code> is given</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO: how does the bootloader inform the kernel where to find initrd? <a href="https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image" class="bare">https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image</a></p>
</div>
<div class="sect2">
<h3 id="initrd-in-desktop-distros"><a class="anchor" href="#initrd-in-desktop-distros"></a><a class="link" href="#initrd-in-desktop-distros">8.1. initrd in desktop distros</a></h3>
<div class="paragraph">
<p>Most modern desktop distributions have an initrd in their root disk to do early setup.</p>
</div>
<div class="paragraph">
<p>The rationale for this is described at: <a href="https://en.wikipedia.org/wiki/Initial_ramdisk" class="bare">https://en.wikipedia.org/wiki/Initial_ramdisk</a></p>
</div>
<div class="paragraph">
<p>One obvious use case is having an encrypted root filesystem: you keep the initrd in an unencrypted partition, and then setup decryption from there.</p>
</div>
<div class="paragraph">
<p>I think GRUB then knows read common disk formats, and then loads that initrd to memory with a <code>/boot/grub/grub.cfg</code> directive of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>initrd /initrd.img-4.4.0-108-generic</pre>
</div>
</div>
<div class="paragraph">
<p>Related: <a href="https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel" class="bare">https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel</a></p>
</div>
</div>
<div class="sect2">
<h3 id="initramfs"><a class="anchor" href="#initramfs"></a><a class="link" href="#initramfs">8.2. initramfs</a></h3>
<div class="paragraph">
<p>initramfs is just like <a href="#initrd">initrd</a>, but you also glue the image directly to the kernel image itself using the kernel&#8217;s build system.</p>
</div>
<div class="paragraph">
<p>Try it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --initramfs
./build-linux --initramfs
./run --initramfs</pre>
</div>
</div>
<div class="paragraph">
<p>Notice how we had to rebuild the Linux kernel this time around as well after Buildroot, since in that build we will be gluing the CPIO to the kernel image.</p>
</div>
<div class="paragraph">
<p>Now, once again, if we look at the QEMU run command generated, we see all that QEMU needs is the <code>-kernel</code> option, no <code>-drive</code> not even <code>-initrd</code>! Pretty cool:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar run_dir)/run.sh"</pre>
</div>
</div>
<div class="paragraph">
<p>It is also interesting to observe how this increases the size of the kernel image if you do a:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls -lh "$(./getvar linux_image)"</pre>
</div>
</div>
<div class="paragraph">
<p>before and after using initramfs, since the <code>.cpio</code> is now glued to the kernel image.</p>
</div>
<div class="paragraph">
<p>Don&#8217;t forget that to stop using initramfs, you must rebuild the kernel without <code>--initramfs</code> to get rid of the attached CPIO image:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux
./run</pre>
</div>
</div>
<div class="paragraph">
<p>Alternatively, consider using <a href="#linux-kernel-build-variants">Linux kernel build variants</a> if you need to switch between initramfs and non initramfs often:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --initramfs
./build-linux --initramfs --linux-build-id initramfs
./run --initramfs --linux-build-id</pre>
</div>
</div>
<div class="paragraph">
<p>Setting up initramfs is very easy: our scripts just set <code>CONFIG_INITRAMFS_SOURCE</code> to point to the CPIO path.</p>
</div>
<div class="paragraph">
<p><a href="http://nairobi-embedded.org/initramfs_tutorial.html" class="bare">http://nairobi-embedded.org/initramfs_tutorial.html</a> shows a full manual setup.</p>
</div>
</div>
<div class="sect2">
<h3 id="rootfs"><a class="anchor" href="#rootfs"></a><a class="link" href="#rootfs">8.3. rootfs</a></h3>
<div class="paragraph">
<p>This is how <code>/proc/mounts</code> shows the root filesystem:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>hard disk: <code>/dev/root on / type ext2 (rw,relatime,block_validity,barrier,user_xattr)</code>. That file does not exist however.</p>
</li>
<li>
<p>initrd: <code>rootfs on / type rootfs (rw)</code></p>
</li>
<li>
<p>initramfs: <code>rootfs on / type rootfs (rw)</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO: understand <code>/dev/root</code> better:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/295060/why-on-some-linux-systems-does-the-root-filesystem-appear-as-dev-root-instead" class="bare">https://unix.stackexchange.com/questions/295060/why-on-some-linux-systems-does-the-root-filesystem-appear-as-dev-root-instead</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/1213770/how-do-you-determine-the-root-device-if-dev-root-is-missing" class="bare">https://superuser.com/questions/1213770/how-do-you-determine-the-root-device-if-dev-root-is-missing</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="devroot"><a class="anchor" href="#devroot"></a><a class="link" href="#devroot">8.3.1. /dev/root</a></h4>
<div class="paragraph">
<p>See: <a href="#rootfs">Section 8.3, &#8220;rootfs&#8221;</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-initrd"><a class="anchor" href="#gem5-initrd"></a><a class="link" href="#gem5-initrd">8.4. gem5 initrd</a></h3>
<div class="paragraph">
<p>TODO we were not able to get it working yet: <a href="https://stackoverflow.com/questions/49261801/how-to-boot-the-linux-kernel-with-initrd-or-initramfs-with-gem5" class="bare">https://stackoverflow.com/questions/49261801/how-to-boot-the-linux-kernel-with-initrd-or-initramfs-with-gem5</a></p>
</div>
<div class="paragraph">
<p>This would require gem5 to load the CPIO into memory, just like QEMU. Grepping <code>initrd</code> shows some ARM hits under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/arch/arm/linux/atag.hh</pre>
</div>
</div>
<div class="paragraph">
<p>but they are commented out.</p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-initramfs"><a class="anchor" href="#gem5-initramfs"></a><a class="link" href="#gem5-initramfs">8.5. gem5 initramfs</a></h3>
<div class="paragraph">
<p>This could in theory be easier to make work than initrd since the emulator does not have to do anything special.</p>
</div>
<div class="paragraph">
<p>However, it didn&#8217;t: boot fails at the end because it does not see the initramfs, but rather tries to open our dummy root filesystem, which unsurprisingly does not have a format in a way that the kernel understands:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>VFS: Cannot open root device "sda" or unknown-block(8,0): error -5</pre>
</div>
</div>
<div class="paragraph">
<p>We think that this might be because gem5 boots directly <code>vmlinux</code>, and not from the final compressed images that contain the attached rootfs such as <code>bzImage</code>, which is what QEMU does, see also: <a href="#vmlinux-vs-bzimage-vs-zimage-vs-image">Section 17.20.1, &#8220;vmlinux vs bzImage vs zImage vs Image&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>To do this failed test, we automatically pass a dummy disk image as of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91 since the scripts don&#8217;t handle a missing <code>--disk-image</code> well, much like is currently done for <a href="#baremetal">Baremetal</a>.</p>
</div>
<div class="paragraph">
<p>Interestingly, using initramfs significantly slows down the gem5 boot, even though it did not work. For example, we&#8217;ve observed a 4x slowdown of as 17062a2e8b6e7888a14c3506e9415989362c58bf for aarch64. This must be because expanding the large attached CPIO must be expensive. We can clearly see from the kernel logs that the kernel just hangs at a point after the message <code>PCI: CLS 0 bytes, default 64</code> for a long time before proceeding further.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="device-tree"><a class="anchor" href="#device-tree"></a><a class="link" href="#device-tree">9. Device tree</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>The device tree is a Linux kernel defined data structure that serves to inform the kernel how the hardware is setup.</p>
</div>
<div class="paragraph">
<p>Device trees serve to reduce the need for hardware vendors to patch the kernel: they just provide a device tree file instead, which is much simpler.</p>
</div>
<div class="paragraph">
<p>x86 does not use it device trees, but many other archs to, notably ARM.</p>
</div>
<div class="paragraph">
<p>This is notably because ARM boards:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>typically don&#8217;t have discoverable hardware extensions like PCI, but rather just put everything on an SoC with magic register addresses</p>
</li>
<li>
<p>are made by a wide variety of vendors due to ARM&#8217;s licensing business model, which increases variability</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The Linux kernel itself has several device trees under <code>./arch/&lt;arch&gt;/boot/dts</code>, see also: <a href="https://stackoverflow.com/questions/21670967/how-to-compile-dts-linux-device-tree-source-files-to-dtb/42839737#42839737" class="bare">https://stackoverflow.com/questions/21670967/how-to-compile-dts-linux-device-tree-source-files-to-dtb/42839737#42839737</a></p>
</div>
<div class="sect2">
<h3 id="dtb-files"><a class="anchor" href="#dtb-files"></a><a class="link" href="#dtb-files">9.1. DTB files</a></h3>
<div class="paragraph">
<p>Files that contain device trees have the <code>.dtb</code> extension when compiled, and <code>.dts</code> when in text form.</p>
</div>
<div class="paragraph">
<p>You can convert between those formats with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>"$(./getvar buildroot_host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
"$(./getvar buildroot_host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts</pre>
</div>
</div>
<div class="paragraph">
<p>Buildroot builds the tool due to <code>BR2_PACKAGE_HOST_DTC=y</code>.</p>
</div>
<div class="paragraph">
<p>On Ubuntu 18.04, the package is named:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install device-tree-compiler</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/14000736/tool-to-visualize-the-device-tree-file-dtb-used-by-the-linux-kernel/39931834#39931834" class="bare">https://stackoverflow.com/questions/14000736/tool-to-visualize-the-device-tree-file-dtb-used-by-the-linux-kernel/39931834#39931834</a></p>
</div>
<div class="paragraph">
<p>Device tree files are provided to the emulator just like the root filesystem and the Linux kernel image.</p>
</div>
<div class="paragraph">
<p>In real hardware, those components are also often provided separately. For example, on the Raspberry Pi 2, the SD card must contain two partitions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the first contains all magic files, including the Linux kernel and the device tree</p>
</li>
<li>
<p>the second contains the root filesystem</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/29837892/how-to-run-a-c-program-with-no-os-on-the-raspberry-pi/40063032#40063032" class="bare">https://stackoverflow.com/questions/29837892/how-to-run-a-c-program-with-no-os-on-the-raspberry-pi/40063032#40063032</a></p>
</div>
</div>
<div class="sect2">
<h3 id="device-tree-syntax"><a class="anchor" href="#device-tree-syntax"></a><a class="link" href="#device-tree-syntax">9.2. Device tree syntax</a></h3>
<div class="paragraph">
<p>Good format descriptions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.raspberrypi.org/documentation/configuration/device-tree.md" class="bare">https://www.raspberrypi.org/documentation/configuration/device-tree.md</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Minimal example</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dts-v1/;

/ {
    a;
};</pre>
</div>
</div>
<div class="paragraph">
<p>Check correctness with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dtc a.dts</pre>
</div>
</div>
<div class="paragraph">
<p>Separate nodes are simply merged by node path, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dts-v1/;

/ {
    a;
};

/ {
    b;
};</pre>
</div>
</div>
<div class="paragraph">
<p>then <code>dtc a.dts</code> gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dts-v1/;

/ {
        a;
        b;
};</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="get-device-tree-from-a-running-kernel"><a class="anchor" href="#get-device-tree-from-a-running-kernel"></a><a class="link" href="#get-device-tree-from-a-running-kernel">9.3. Get device tree from a running kernel</a></h3>
<div class="paragraph">
<p><a href="https://unix.stackexchange.com/questions/265890/is-it-possible-to-get-the-information-for-a-device-tree-using-sys-of-a-running/330926#330926" class="bare">https://unix.stackexchange.com/questions/265890/is-it-possible-to-get-the-information-for-a-device-tree-using-sys-of-a-running/330926#330926</a></p>
</div>
<div class="paragraph">
<p>This is specially interesting because QEMU and gem5 are capable of generating DTBs that match the selected machine depending on dynamic command line parameters for some types of machines.</p>
</div>
<div class="paragraph">
<p>So observing the device tree from the guest allows to easily see what the emulator has generated.</p>
</div>
<div class="paragraph">
<p>Compile the <code>dtc</code> tool into the root filesystem:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --arch aarch64 \
  --config 'BR2_PACKAGE_DTC=y' \
  --config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
;</pre>
</div>
</div>
<div class="paragraph">
<p><code>-M virt</code> for example, which we use by default for <code>aarch64</code>, boots just fine without the <code>-dtb</code> option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>Then, from inside the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dtc -I fs -O dts /sys/firmware/devicetree/base</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>        cpus {
                #address-cells = &lt;0x1&gt;;
                #size-cells = &lt;0x0&gt;;

                cpu@0 {
                        compatible = "arm,cortex-a57";
                        device_type = "cpu";
                        reg = &lt;0x0&gt;;
                };
        };</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="device-tree-emulator-generation"><a class="anchor" href="#device-tree-emulator-generation"></a><a class="link" href="#device-tree-emulator-generation">9.4. Device tree emulator generation</a></h3>
<div class="paragraph">
<p>Since emulators know everything about the hardware, they can automatically generate device trees for us, which is very convenient.</p>
</div>
<div class="paragraph">
<p>This is the case for both QEMU and gem5.</p>
</div>
<div class="paragraph">
<p>For example, if we increase the <a href="#number-of-cores">number of cores</a> to 2:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --cpus 2</pre>
</div>
</div>
<div class="paragraph">
<p>QEMU automatically adds a second CPU to the DTB!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>                cpu@0 {
                cpu@1 {</pre>
</div>
</div>
<div class="paragraph">
<p>The action seems to be happening at: <code>hw/arm/virt.c</code>.</p>
</div>
<div class="paragraph">
<p>You can dump the DTB QEMU generated with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 -- -machine dumpdtb=dtb.dtb</pre>
</div>
</div>
<div class="paragraph">
<p>as mentioned at: <a href="https://lists.gnu.org/archive/html/qemu-discuss/2017-02/msg00051.html" class="bare">https://lists.gnu.org/archive/html/qemu-discuss/2017-02/msg00051.html</a></p>
</div>
<div class="paragraph">
<p><a href="#gem5-fs-biglittle">gem5 fs_bigLITTLE</a> 2a9573f5942b5416fb0570cf5cb6cdecba733392 can also generate its own DTB.</p>
</div>
<div class="paragraph">
<p>gem5 can generate DTBs on ARM with <code>--generate-dtb</code>. The generated DTB is placed in the <a href="#m5out-directory">m5out directory</a> named as <code>system.dtb</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="kvm"><a class="anchor" href="#kvm"></a><a class="link" href="#kvm">10. KVM</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine">KVM</a> is Linux kernel interface that <a href="#benchmark-linux-kernel-boot">greatly speeds up</a> execution of virtual machines.</p>
</div>
<div class="paragraph">
<p>You can make QEMU or <a href="#gem5-kvm">gem5</a> by passing enabling KVM with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kvm</pre>
</div>
</div>
<div class="paragraph">
<p>KVM works by running userland instructions natively directly on the real hardware instead of running a software simulation of those instructions.</p>
</div>
<div class="paragraph">
<p>Therefore, KVM only works if you the host architecture is the same as the guest architecture. This means that this will likely only work for x86 guests since almost all development machines are x86 nowadays. Unless you are <a href="https://www.youtube.com/watch?v=8ItXpmLsINs">running an ARM desktop for some weird reason</a> :-)</p>
</div>
<div class="paragraph">
<p>We don&#8217;t enable KVM by default because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>it limits visibility, since more things are running natively:</p>
<div class="ulist">
<ul>
<li>
<p>can&#8217;t use <a href="#gdb">GDB</a></p>
</li>
<li>
<p>can&#8217;t do <a href="#tracing">instruction tracing</a></p>
</li>
<li>
<p>on gem5, you lose <a href="#gem5-run-benchmark">cycle counts</a> and therefor any notion of performance</p>
</li>
</ul>
</div>
</li>
<li>
<p>QEMU kernel boots are already <a href="#benchmark-linux-kernel-boot">fast enough</a> for most purposes without it</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>One important use case for KVM is to fast forward gem5 execution, often to skip boot, take a <a href="#gem5-checkpoint">gem5 checkpoint</a>, and then move on to a more detailed and slow simulation</p>
</div>
<div class="sect2">
<h3 id="kvm-arm"><a class="anchor" href="#kvm-arm"></a><a class="link" href="#kvm-arm">10.1. KVM arm</a></h3>
<div class="paragraph">
<p>TODO: we haven&#8217;t gotten it to work yet, but it should be doable, and this is an outline of how to do it. Just don&#8217;t expect this to tested very often for now.</p>
</div>
<div class="paragraph">
<p>We can test KVM on arm by running this repository inside an Ubuntu arm QEMU VM.</p>
</div>
<div class="paragraph">
<p>This produces no speedup of course, since the VM is already slow since it cannot use KVM on the x86 host.</p>
</div>
<div class="paragraph">
<p>First, obtain an Ubuntu arm64 virtual machine as explained at: <a href="https://askubuntu.com/questions/281763/is-there-any-prebuilt-qemu-ubuntu-image32bit-online/1081171#1081171" class="bare">https://askubuntu.com/questions/281763/is-there-any-prebuilt-qemu-ubuntu-image32bit-online/1081171#1081171</a></p>
</div>
<div class="paragraph">
<p>Then, from inside that image:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install git
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./setup -y</pre>
</div>
</div>
<div class="paragraph">
<p>and then proceed exactly as in <a href="#prebuilt">Prebuilt setup</a>.</p>
</div>
<div class="paragraph">
<p>We don&#8217;t want to build the full Buildroot image inside the VM as that would be way too slow, thus the recommendation for the prebuilt setup.</p>
</div>
<div class="paragraph">
<p>TODO: do the right thing and cross compile QEMU and gem5. gem5&#8217;s Python parts might be a pain. QEMU should be easy: <a href="https://stackoverflow.com/questions/26514252/cross-compile-qemu-for-arm" class="bare">https://stackoverflow.com/questions/26514252/cross-compile-qemu-for-arm</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-kvm"><a class="anchor" href="#gem5-kvm"></a><a class="link" href="#gem5-kvm">10.2. gem5 KVM</a></h3>
<div class="paragraph">
<p>While gem5 does have KVM, as of 2019 its support has not been very good, because debugging it is harder and people haven&#8217;t focused intensively on it.</p>
</div>
<div class="paragraph">
<p>X86 was broken with pending patches: <a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15046.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15046.html</a> It failed immediately on:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)</pre>
</div>
</div>
<div class="paragraph">
<p>also mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/62687463/gem5-kvm-doesnt-work-with-error-0x80000021" class="bare">https://stackoverflow.com/questions/62687463/gem5-kvm-doesnt-work-with-error-0x80000021</a></p>
</li>
<li>
<p><a href="https://gem5-users.gem5.narkive.com/8DBihuUx/running-fs-py-with-x86kvmcpu-failed" class="bare">https://gem5-users.gem5.narkive.com/8DBihuUx/running-fs-py-with-x86kvmcpu-failed</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ARM thread: <a href="https://stackoverflow.com/questions/53523087/how-to-run-gem5-on-kvm-on-arm-with-multiple-cores" class="bare">https://stackoverflow.com/questions/53523087/how-to-run-gem5-on-kvm-on-arm-with-multiple-cores</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="user-mode-simulation"><a class="anchor" href="#user-mode-simulation"></a><a class="link" href="#user-mode-simulation">11. User mode simulation</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Both QEMU and gem5 have an user mode simulation mode in addition to full system simulation that we consider elsewhere in this project.</p>
</div>
<div class="paragraph">
<p>In QEMU, it is called just <a href="#qemu-user-mode-getting-started">"user mode"</a>, and in gem5 it is called <a href="#gem5-syscall-emulation-mode">syscall emulation mode</a>.</p>
</div>
<div class="paragraph">
<p>In both, the basic idea is the same.</p>
</div>
<div class="paragraph">
<p>User mode simulation takes regular userland executables of any arch as input and executes them directly, without booting a kernel.</p>
</div>
<div class="paragraph">
<p>Instead of simulating the full system, it translates normal instructions like in full system mode, but magically forwards system calls to the host OS.</p>
</div>
<div class="paragraph">
<p>Advantages over full system simulation:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the simulation may <a href="#user-mode-vs-full-system-benchmark">run faster</a> since you don&#8217;t have to simulate the Linux kernel and several device models</p>
</li>
<li>
<p>you don&#8217;t need to build your own kernel or root filesystem, which saves time. You still need a toolchain however, but the pre-packaged ones may work fine.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Disadvantages:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>lower guest to host portability:</p>
<div class="ulist">
<ul>
<li>
<p>TODO confirm: host OS == guest OS?</p>
</li>
<li>
<p>TODO confirm: the host Linux kernel should be newer than the kernel the executable was built for.</p>
<div class="paragraph">
<p>It may still work even if that is not the case, but could fail is a missing system call is reached.</p>
</div>
<div class="paragraph">
<p>The target Linux kernel of the executable is a GCC toolchain build-time configuration.</p>
</div>
</li>
<li>
<p>emulator implementers have to keep up with libc changes, some of which break even a C hello world due setup code executed before main.</p>
<div class="paragraph">
<p>See also: <a href="#user-mode-simulation-with-glibc">Section 11.4, &#8220;User mode simulation with glibc&#8221;</a></p>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>cannot be used to test the Linux kernel or any devices, and results are less representative of a real system since we are faking more</p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="qemu-user-mode-getting-started"><a class="anchor" href="#qemu-user-mode-getting-started"></a><a class="link" href="#qemu-user-mode-getting-started">11.1. QEMU user mode getting started</a></h3>
<div class="paragraph">
<p>Let&#8217;s run <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/command_line_arguments.c">userland/c/command_line_arguments.c</a> built with the Buildroot toolchain on QEMU user mode:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build user-mode-qemu
./run \
  --userland userland/c/command_line_arguments.c \
  --cli-args='asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/command_line_arguments.out
asdf
qw er</pre>
</div>
</div>
<div class="paragraph">
<p><code>./run --userland</code> path resolution is analogous to <a href="#baremetal-setup-getting-started">that of <code>./run --baremetal</code></a>.</p>
</div>
<div class="paragraph">
<p><code>./build user-mode-qemu</code> first builds Buildroot, and then runs <code>./build-userland</code>, which is further documented at: <a href="#userland-setup">Section 2.8, &#8220;Userland setup&#8221;</a>. It also builds QEMU. If you ahve already done a <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a> previously, this will be very fast.</p>
</div>
<div class="paragraph">
<p>If you modify the userland programs, rebuild simply with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland</pre>
</div>
</div>
<div class="paragraph">
<p>To rebuild just QEMU userland if you hack it, use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-qemu --mode userland</pre>
</div>
</div>
<div class="paragraph">
<p>The:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--mode userland</pre>
</div>
</div>
<div class="paragraph">
<p>is needed because QEMU has two separate executables:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>qemu-x86_64</code> for userland</p>
</li>
<li>
<p><code>qemu-system-x86_64</code> for full system</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="user-mode-gdb"><a class="anchor" href="#user-mode-gdb"></a><a class="link" href="#user-mode-gdb">11.1.1. User mode GDB</a></h4>
<div class="paragraph">
<p>It&#8217;s nice when <a href="#gdb">the obvious</a> just works, right?</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and on another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb \
  --arch aarch64 \
  --userland userland/c/command_line_arguments.c \
  main \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Or alternatively, if you are using <a href="#tmux">tmux</a>, do everything in one go with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --gdb \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>To stop at the very first instruction of a freestanding program, just use <code>--no-continue</code>. A good example of this is shown at: <a href="#freestanding-programs">Section 28.5.1, &#8220;Freestanding programs&#8221;</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="user-mode-tests"><a class="anchor" href="#user-mode-tests"></a><a class="link" href="#user-mode-tests">11.2. User mode tests</a></h3>
<div class="paragraph">
<p>Automatically run all userland tests that can be run in user mode simulation, and check that they exit with status 0:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --all-archs test-executables-userland
./test-executables --all-archs --all-emulators</pre>
</div>
</div>
<div class="paragraph">
<p>Or just for QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --all-archs test-executables-userland-qemu
./test-executables --all-archs --emulator qemu</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-executables">test-executables</a></p>
</div>
<div class="paragraph">
<p>This script skips a manually configured list of tests, notably:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>tests that depend on a full running kernel and cannot be run in user mode simulation, e.g. those that rely on kernel modules</p>
</li>
<li>
<p>tests that require user interaction</p>
</li>
<li>
<p>tests that take perceptible amounts of time</p>
</li>
<li>
<p>known bugs we didn&#8217;t have time to fix ;-)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tests under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/">userland/libs/</a> are only run if <code>--package</code> or <code>--package-all</code> are given as described at <a href="#userland-libs-directory">userland/libs directory</a>.</p>
</div>
<div class="paragraph">
<p>The gem5 tests require building statically with build id <code>static</code>, see also: <a href="#gem5-syscall-emulation-mode">Section 11.7, &#8220;gem5 syscall emulation mode&#8221;</a>. TODO automate this better.</p>
</div>
<div class="paragraph">
<p>See: <a href="#test-this-repo">Section 38.16, &#8220;Test this repo&#8221;</a> for more useful testing tips.</p>
</div>
</div>
<div class="sect2">
<h3 id="user-mode-buildroot-executables"><a class="anchor" href="#user-mode-buildroot-executables"></a><a class="link" href="#user-mode-buildroot-executables">11.3. User mode Buildroot executables</a></h3>
<div class="paragraph">
<p>If you followed <a href="#qemu-buildroot-setup">QEMU Buildroot setup</a>, you can now run the executables created by Buildroot directly as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --userland "$(./getvar buildroot_target_dir)/bin/echo" \
  --cli-args='asdf' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>To easily explore the userland executable environment interactively, you can do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --terminal \
;</pre>
</div>
</div>
<div class="paragraph">
<p>or:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --cli-args='-c "uname -a &amp;&amp; pwd"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Here is an interesting examples of this: <a href="#linux-test-project">Section 17.19.1, &#8220;Linux Test Project&#8221;</a></p>
</div>
</div>
<div class="sect2">
<h3 id="user-mode-simulation-with-glibc"><a class="anchor" href="#user-mode-simulation-with-glibc"></a><a class="link" href="#user-mode-simulation-with-glibc">11.4. User mode simulation with glibc</a></h3>
<div class="paragraph">
<p>At 125d14805f769104f93c510bedaa685a52ec025d we <a href="#libc-choice">moved Buildroot from uClibc to glibc</a>, and caused some user mode pain, which we document here.</p>
</div>
<div class="sect3">
<h4 id="fatal-kernel-too-old-failure-in-userland-simulation"><a class="anchor" href="#fatal-kernel-too-old-failure-in-userland-simulation"></a><a class="link" href="#fatal-kernel-too-old-failure-in-userland-simulation">11.4.1. FATAL: kernel too old failure in userland simulation</a></h4>
<div class="paragraph">
<p>glibc has a check for kernel version, likely obtained from the <code>uname</code> syscall, and if the kernel is not new enough, it quits.</p>
</div>
<div class="paragraph">
<p>Both gem5 and QEMU however allow setting the reported <code>uname</code> version from the command line, which we do to always match our toolchain.</p>
</div>
<div class="paragraph">
<p>QEMU by default copies the host <code>uname</code> value, but we always override it in our scripts.</p>
</div>
<div class="paragraph">
<p>Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulators don&#8217;t have.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --kernel-version 4.18 --userland userland/posix/uname.c</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/uname.c">userland/posix/uname.c</a>.</p>
</div>
<div class="paragraph">
<p>The QEMU source that does this is at: <a href="https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931" class="bare">https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m" class="bare">https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/53085048/how-to-compile-and-run-an-executable-in-gem5-syscall-emulation-mode-with-se-py/53085049#53085049" class="bare">https://stackoverflow.com/questions/53085048/how-to-compile-and-run-an-executable-in-gem5-syscall-emulation-mode-with-se-py/53085049#53085049</a></p>
</li>
<li>
<p><a href="https://gem5-review.googlesource.com/c/public/gem5/+/15855" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/15855</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The ID is just hardcoded on the source:</p>
</div>
</div>
<div class="sect3">
<h4 id="stack-smashing-detected-when-using-glibc"><a class="anchor" href="#stack-smashing-detected-when-using-glibc"></a><a class="link" href="#stack-smashing-detected-when-using-glibc">11.4.2. stack smashing detected when using glibc</a></h4>
<div class="paragraph">
<p>For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.</p>
</div>
<div class="paragraph">
<p>Other archs work as they different host libc is skipped. <a href="#user-mode-static-executables">User mode static executables</a> also work.</p>
</div>
<div class="paragraph">
<p>We have worked around this with with <a href="https://bugs.launchpad.net/qemu/+bug/1701798/comments/12" class="bare">https://bugs.launchpad.net/qemu/+bug/1701798/comments/12</a> from the thread: <a href="https://bugs.launchpad.net/qemu/+bug/1701798" class="bare">https://bugs.launchpad.net/qemu/+bug/1701798</a> by creating the file: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/ld.so.cache">rootfs_overlay/etc/ld.so.cache</a> which is a symlink to a file that cannot exist: <code>/dev/null/nonexistent</code>.</p>
</div>
<div class="paragraph">
<p>Reproduction:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache"
./run --userland userland/c/hello.c
./run --userland userland/c/hello.c --qemu-which host</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>*** stack smashing detected ***: &lt;unknown&gt; terminated
qemu: uncaught target signal 6 (Aborted) - core dumped</pre>
</div>
</div>
<div class="paragraph">
<p>To get things working again, restore <code>ld.so.cache</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot</pre>
</div>
</div>
<div class="paragraph">
<p>I&#8217;ve also tested on an Ubuntu 16.04 guest and the failure is different one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>qemu: uncaught target signal 4 (Illegal instruction) - core dumped</pre>
</div>
</div>
<div class="paragraph">
<p>A non-QEMU-specific example of stack smashing is shown at: <a href="https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264" class="bare">https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264</a></p>
</div>
<div class="paragraph">
<p>Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="user-mode-static-executables"><a class="anchor" href="#user-mode-static-executables"></a><a class="link" href="#user-mode-static-executables">11.5. User mode static executables</a></h3>
<div class="paragraph">
<p>Example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland \
  --arch aarch64 \
  --static \
;
./run \
  --arch aarch64 \
  --static \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the <code>-L</code> option so that it can find the dynamic linker and shared libraries, see also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/54802670/using-dynamic-linker-with-qemu-arm/64551293#64551293" class="bare">https://stackoverflow.com/questions/54802670/using-dynamic-linker-with-qemu-arm/64551293#64551293</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/khow-to-gdb-step-debug-a-dynamically-linked-executable-in-qemu-user-mode" class="bare">https://stackoverflow.com/questions/khow-to-gdb-step-debug-a-dynamically-linked-executable-in-qemu-user-mode</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We pass <code>-L</code> by default, so everything just works.</p>
</div>
<div class="paragraph">
<p>However, in case something goes wrong, you can also try statically linked executables, since this mechanism tends to be a bit more stable, for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>QEMU x86_64 guest on x86_64 host was failing with <a href="#stack-smashing-detected-when-using-glibc">stack smashing detected when using glibc</a>, but we found a workaround</p>
</li>
<li>
<p>gem5 user only supported static executables in the past, as mentioned at: <a href="#gem5-syscall-emulation-mode">Section 11.7, &#8220;gem5 syscall emulation mode&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Running statically linked executables sometimes makes things break:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#user-mode-static-executables-with-dynamic-libraries">User mode static executables with dynamic libraries</a></p>
</li>
<li>
<p>TODO understand why:</p>
<div class="literalblock">
<div class="content">
<pre>./run --static --userland userland/c/file_write_read.c</pre>
</div>
</div>
<div class="paragraph">
<p>fails our assertion that the data was read back correctly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Assertion `strcmp(data, output) == 0' faile</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="user-mode-static-executables-with-dynamic-libraries"><a class="anchor" href="#user-mode-static-executables-with-dynamic-libraries"></a><a class="link" href="#user-mode-static-executables-with-dynamic-libraries">11.5.1. User mode static executables with dynamic libraries</a></h4>
<div class="paragraph">
<p>One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).</p>
</div>
<div class="paragraph">
<p>So programs that rely on those libraries might not compile as GCC can&#8217;t find the <code>.a</code> version of the library.</p>
</div>
<div class="paragraph">
<p>For example, if we try to build <a href="#blas">BLAS</a> statically:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --package openblas --static -- userland/libs/openblas/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>it fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ld: cannot find -lopenblas</pre>
</div>
</div>
<div class="sect4">
<h5 id="cpp-static-and-pthreads"><a class="anchor" href="#cpp-static-and-pthreads"></a><a class="link" href="#cpp-static-and-pthreads">11.5.1.1. C++ static and pthreads</a></h5>
<div class="paragraph">
<p><code>g++</code> and pthreads also causes issues:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why" class="bare">https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/58848694/gcc-whole-archive-recipe-for-static-linking-to-pthread-stopped-working-in-rec" class="bare">https://stackoverflow.com/questions/58848694/gcc-whole-archive-recipe-for-static-linking-to-pthread-stopped-working-in-rec</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>As a consequence, the following just hangs as of LKMC ca0403849e03844a328029d70c08556155dc1cd0 + 1 the example <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/std_atomic.cpp">userland/cpp/atomic/std_atomic.cpp</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland userland/cpp/atomic/std_atomic.cpp --static</pre>
</div>
</div>
<div class="paragraph">
<p>And before that, it used to fail with other randomly different errors, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.
qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.</pre>
</div>
</div>
<div class="paragraph">
<p>And a native Ubuntu 18.04 AMD64 run with static compilation segfaults.</p>
</div>
<div class="paragraph">
<p>As of LKMC f5d4998ff51a548ed3f5153aacb0411d22022058 the aarch64 error:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --userland userland/cpp/atomic/fail.cpp --static</pre>
</div>
</div>
<div class="paragraph">
<p>is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error 16781344
qemu: uncaught target signal 6 (Aborted) - core dumped</pre>
</div>
</div>
<div class="paragraph">
<p>The workaround:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-pthread -Wl,--whole-archive -lpthread -Wl,--no-whole-archive</pre>
</div>
</div>
<div class="paragraph">
<p>fixes some of the problems, but not all TODO which were missing?, so we are just skipping those tests for now.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="syscall-emulation-mode-program-stdin"><a class="anchor" href="#syscall-emulation-mode-program-stdin"></a><a class="link" href="#syscall-emulation-mode-program-stdin">11.6. syscall emulation mode program stdin</a></h3>
<div class="paragraph">
<p>The following work on both QEMU and gem5 as of LKMC 99d6bc6bc19d4c7f62b172643be95d9c43c26145 + 1. Interactive input:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland userland/c/getchar.c</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/getchar.c">userland/c/getchar.c</a></p>
</div>
<div class="paragraph">
<p>A line of type should show:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>enter a character:</pre>
</div>
</div>
<div class="paragraph">
<p>and after pressing say <code>a</code> and Enter, we get:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>you entered: a</pre>
</div>
</div>
<div class="paragraph">
<p>Note however that due to <a href="#qemu-user-mode-does-not-show-stdout-immediately">QEMU user mode does not show stdout immediately</a> we don&#8217;t really see the initial <code>enter a character</code> line.</p>
</div>
<div class="paragraph">
<p>Non-interactive input from a file by forwarding emulators stdin implicitly through our Python scripts:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf a &gt; f.tmp
./run --userland userland/c/getchar.c &lt; f.tmp</pre>
</div>
</div>
<div class="paragraph">
<p>Input from a file by explicitly requesting our scripts to use it via the Python API:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf a &gt; f.tmp
./run --emulator gem5 --userland userland/c/getchar.c --stdin-file f.tmp</pre>
</div>
</div>
<div class="paragraph">
<p>This is especially useful when running tests that require stdin input.</p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-syscall-emulation-mode"><a class="anchor" href="#gem5-syscall-emulation-mode"></a><a class="link" href="#gem5-syscall-emulation-mode">11.7. gem5 syscall emulation mode</a></h3>
<div class="paragraph">
<p>Less robust than QEMU&#8217;s, but still usable:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program" class="bare">https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>There are much more unimplemented syscalls in gem5 than in QEMU. Many of those are trivial to implement however.</p>
</div>
<div class="paragraph">
<p>So let&#8217;s just play with some static ones:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --arch aarch64
./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: how to escape spaces on the command line arguments?</p>
</div>
<div class="paragraph">
<p><a href="#user-mode-gdb">GDB step debug</a> also works normally on gem5:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;
./run-gdb \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  main \
;</pre>
</div>
</div>
<div class="sect3">
<h4 id="gem5-dynamic-linked-executables-in-syscall-emulation"><a class="anchor" href="#gem5-dynamic-linked-executables-in-syscall-emulation"></a><a class="link" href="#gem5-dynamic-linked-executables-in-syscall-emulation">11.7.1. gem5 dynamic linked executables in syscall emulation</a></h4>
<div class="paragraph">
<p>Support for dynamic linking was added in November 2019:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5/50696098#50696098" class="bare">https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5/50696098#50696098</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/64547306/cannot-open-lib-ld-linux-aarch64-so-1-in-qemu-or-gem5/64551313#64551313" class="bare">https://stackoverflow.com/questions/64547306/cannot-open-lib-ld-linux-aarch64-so-1-in-qemu-or-gem5/64551313#64551313</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Note that as shown at <a href="#benchmark-emulators-on-userland-executables">Section 35.2.2, &#8220;Benchmark emulators on userland executables&#8221;</a>, the dynamic version runs 200x more instructions, which might have an impact on smaller simulations in detailed CPUs.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-syscall-emulation-exit-status"><a class="anchor" href="#gem5-syscall-emulation-exit-status"></a><a class="link" href="#gem5-syscall-emulation-exit-status">11.7.2. gem5 syscall emulation exit status</a></h4>
<div class="paragraph">
<p>As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy <code>se.py</code> script does not forward the exit status of syscall emulation mode, you can test it with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --dry-run --emulator gem5 --userland userland/c/false.c</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/false.c">userland/c/false.c</a>.</p>
</div>
<div class="paragraph">
<p>Then manually run the generated gem5 CLI, and do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>and the output is always <code>0</code>.</p>
</div>
<div class="paragraph">
<p>Instead, it just outputs a message to stdout just like for <a href="#m5-fail">m5 fail</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Simulated exit code not 0! Exit code is 1</pre>
</div>
</div>
<div class="paragraph">
<p>which we parse in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> and then exit with the correct result ourselves&#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>Related thread: <a href="https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully" class="bare">https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-syscall-emulation-mode-syscall-tracing"><a class="anchor" href="#gem5-syscall-emulation-mode-syscall-tracing"></a><a class="link" href="#gem5-syscall-emulation-mode-syscall-tracing">11.7.3. gem5 syscall emulation mode syscall tracing</a></h4>
<div class="paragraph">
<p>Since gem5 has to implement syscalls itself in syscall emulation mode, it can of course clearly see which syscalls are being made, and we can log them for debug purposes with <a href="#gem5-tracing">gem5 tracing</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --emulator gem5 \
  --userland userland/arch/x86_64/freestanding/linux/hello.S \
  --trace-stdout \
  --trace ExecAll,SyscallBase,SyscallVerbose \
;</pre>
</div>
</div>
<div class="paragraph">
<p>the trace as of f2eeceb1cde13a5ff740727526bf916b356cee38 + 1 contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu A0 T0 : @asm_main_after_prologue    : mov   rdi, 0x1
      0: system.cpu A0 T0 : @asm_main_after_prologue.0  :   MOV_R_I : limm   rax, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7    : mov rdi, 0x1
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7.0  :   MOV_R_I : limm   rdi, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14    : lea        rsi, DS:[rip + 0x19]
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14.0  :   LEA_R_P : rdip   t7, %ctrl153,  : IntAlu :  D=0x000000000040008d  flags=(IsInteger|IsMicroop|IsDelayedCommit|IsFirstMicroop)
   2500: system.cpu A0 T0 : @asm_main_after_prologue+14.1  :   LEA_R_P : lea   rsi, DS:[t7 + 0x19] : IntAlu :  D=0x00000000004000a6  flags=(IsInteger|IsMicroop|IsLastMicroop)
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21    : mov        rdi, 0x6
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21.0  :   MOV_R_I : limm   rdx, 0x6 : IntAlu :  D=0x0000000000000006  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   4000: system.cpu: T0 : syscall write called w/arguments 1, 4194470, 6, 0, 0, 0
hello
   4000: system.cpu: T0 : syscall write returns 6
   4000: system.cpu A0 T0 : @asm_main_after_prologue+28    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30    : mov        rdi, 0x3c
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30.0  :   MOV_R_I : limm   rax, 0x3c : IntAlu :  D=0x000000000000003c  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37    : mov        rdi, 0
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37.0  :   MOV_R_I : limm   rdi, 0  : IntAlu :  D=0x0000000000000000  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6500: system.cpu: T0 : syscall exit called w/arguments 0, 4194470, 6, 0, 0, 0
   6500: system.cpu: T0 : syscall exit returns 0
   6500: system.cpu A0 T0 : @asm_main_after_prologue+44    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that two syscall lines were added for each syscall, showing the syscall inputs and exit status, just like a mini <code>strace</code>!</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-syscall-emulation-multithreading"><a class="anchor" href="#gem5-syscall-emulation-multithreading"></a><a class="link" href="#gem5-syscall-emulation-multithreading">11.7.4. gem5 syscall emulation multithreading</a></h4>
<div class="paragraph">
<p>gem5 user mode multithreading has been particularly flaky compared <a href="#qemu-user-mode-multithreading">to QEMU&#8217;s</a>, but work is being put into improving it.</p>
</div>
<div class="paragraph">
<p>In gem5 syscall simulation, the <code>fork</code> syscall checks if there is a free CPU, and if there is a free one, the new threads runs on that CPU.</p>
</div>
<div class="paragraph">
<p>Otherwise, the <code>fork</code> call, and therefore higher level interfaces to <code>fork</code> such as <code>pthread_create</code> also fail and return a failure return status in the guest.</p>
</div>
<div class="paragraph">
<p>For example, if we use just one CPU for <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_self.c">userland/posix/pthread_self.c</a> which spawns one thread besides <code>main</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 1 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1</pre>
</div>
</div>
<div class="paragraph">
<p>fails with this error message coming from the guest stderr:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pthread_create: Resource temporarily unavailable</pre>
</div>
</div>
<div class="paragraph">
<p>It works however if we add on extra CPU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1</pre>
</div>
</div>
<div class="paragraph">
<p>Once threads exit, their CPU is freed and becomes available for new <code>fork</code> calls: For example, the following run spawns a thread, joins it, and then spawns again, and 2 CPUs are enough:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args '1 2'</pre>
</div>
</div>
<div class="paragraph">
<p>because at each point in time, only up to two threads are running.</p>
</div>
<div class="paragraph">
<p>gem5 syscall emulation does show the expected number of cores when queried, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 1 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5
./run --cpus 2 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>outputs <code>1</code> and <code>2</code> respectively.</p>
</div>
<div class="paragraph">
<p>This can also be clearly by running <code>sched_getcpu</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --cli-args  4 \
  --cpus 8 \
  --emulator gem5 \
  --userland userland/linux/sched_getcpu.c \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which necessarily produces an output containing the CPU numbers from 1 to 4 and no higher:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>1
3
4
2</pre>
</div>
</div>
<div class="paragraph">
<p>TODO why does the <code>2</code> come at the end here? Would be good to do a detailed assembly run analysis.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-syscall-emulation-multiple-executables"><a class="anchor" href="#gem5-syscall-emulation-multiple-executables"></a><a class="link" href="#gem5-syscall-emulation-multiple-executables">11.7.5. gem5 syscall emulation multiple executables</a></h4>
<div class="paragraph">
<p>gem5 syscall emulation has the nice feature of allowing you to run multiple executables "at once".</p>
</div>
<div class="paragraph">
<p>Each executable starts running on the next free core much as if it had been forked right at the start of simulation: <a href="#gem5-syscall-emulation-multithreading">gem5 syscall emulation multithreading</a>.</p>
</div>
<div class="paragraph">
<p>This can be useful to quickly create deterministic multi-CPU workload.</p>
</div>
<div class="paragraph">
<p><code>se.py --cmd</code> takes a semicolon separated list, so we could do which LKMC exposes this by taking <code>--userland</code> multiple times as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --cpus 2 \
  --emulator gem5 \
  --userland userland/posix/getpid.c \
  --userland userland/posix/getpid.c \
;</pre>
</div>
</div>
<div class="paragraph">
<p>We need at least one CPU per executable, just like when forking new processes.</p>
</div>
<div class="paragraph">
<p>The outcome of this is that we see two different <code>pid</code> messages printed to stdout:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pid=101
pid=100</pre>
</div>
</div>
<div class="paragraph">
<p>since from <a href="#gem5-process">gem5 <code>Process</code></a> we can see that se.py sets up one different PID per executable starting at 100:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    workloads = options.cmd.split(';')
    idx = 0
    for wrkld in workloads:
        process = Process(pid = 100 + idx)</pre>
</div>
</div>
<div class="paragraph">
<p>We can also see that these processes are running concurrently with <a href="#gem5-tracing">gem5 tracing</a> by hacking:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  --debug-flags ExecAll \
  --debug-file cout \</pre>
</div>
</div>
<div class="paragraph">
<p>which starts with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu1: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
      0: system.cpu0: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
    500: system.cpu0: A0 T0 : @__end__+274873647044    :   bl   &lt;__end__+274873649648&gt; : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)
    500: system.cpu1: A0 T0 : @__end__+274873647044    :   bl   &lt;__end__+274873649648&gt; : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)</pre>
</div>
</div>
<div class="paragraph">
<p>and therefore shows one instruction running on each CPU for each process at the same time.</p>
</div>
<div class="sect4">
<h5 id="gem5-syscall-emulation-smt"><a class="anchor" href="#gem5-syscall-emulation-smt"></a><a class="link" href="#gem5-syscall-emulation-smt">11.7.5.1. gem5 syscall emulation --smt</a></h5>
<div class="paragraph">
<p>gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4 syscall emulation has an <code>--smt</code> option presumably for <a href="#hardware-threads">Hardware threads</a> but it has been neglected forever it seems: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/104" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/104</a></p>
</div>
<div class="paragraph">
<p>If we start from the manually hacked working command from <a href="#gem5-syscall-emulation-multiple-executables">gem5 syscall emulation multiple executables</a> and try to add:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--cpu 1 --cpu-type Derivo3CPU --caches</pre>
</div>
</div>
<div class="paragraph">
<p>We choose <a href="#gem5-derivo3cpu"><code>DerivO3CPU</code></a> because of the se.py assert:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>example/se.py:115:        assert(options.cpu_type == "DerivO3CPU")</pre>
</div>
</div>
<div class="paragraph">
<p>But then that fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gem5.opt: /path/to/linux-kernel-module-cheat/out/gem5/master3/build/ARM/cpu/o3/cpu.cc:205: FullO3CPU&lt;Impl&gt;::FullO3CPU(DerivO3CPUParams*) [with Impl = O3CPUImpl]: Assertion `params-&gt;numPhysVecPredRegs &gt;= numThreads * TheISA::NumVecPredRegs' failed.
Program aborted at tick 0</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qemu-user-mode-quirks"><a class="anchor" href="#qemu-user-mode-quirks"></a><a class="link" href="#qemu-user-mode-quirks">11.8. QEMU user mode quirks</a></h3>
<div class="sect3">
<h4 id="qemu-user-mode-does-not-show-stdout-immediately"><a class="anchor" href="#qemu-user-mode-does-not-show-stdout-immediately"></a><a class="link" href="#qemu-user-mode-does-not-show-stdout-immediately">11.8.1. QEMU user mode does not show stdout immediately</a></h4>
<div class="paragraph">
<p>At 8d8307ac0710164701f6e14c99a69ee172ccbb70 + 1, I noticed that if you run <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count.c">userland/posix/count.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland userland/posix/count_to.c --cli-args 3</pre>
</div>
</div>
<div class="paragraph">
<p>it first waits for 3 seconds, then the program exits, and then it dumps all the stdout at once, instead of counting once every second as expected.</p>
</div>
<div class="paragraph">
<p>The same can be reproduced by copying the raw QEMU command and piping it through <code>tee</code>, so I don&#8217;t think it is a bug in our setup:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/path/to/linux-kernel-module-cheat/out/qemu/default/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/posix/count.out \
  3 \
| tee</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: investigate further and then possibly post on QEMU mailing list.</p>
</div>
<div class="sect4">
<h5 id="qemu-user-mode-does-not-show-errors"><a class="anchor" href="#qemu-user-mode-does-not-show-errors"></a><a class="link" href="#qemu-user-mode-does-not-show-errors">11.8.1.1. QEMU user mode does not show errors</a></h5>
<div class="paragraph">
<p>Similarly to <a href="#qemu-user-mode-does-not-show-stdout-immediately">QEMU user mode does not show stdout immediately</a>, QEMU error messages do not show at all through pipes.</p>
</div>
<div class="paragraph">
<p>In particular, it does not say anything if you pass it a non-existing executable:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>qemu-x86_64 asdf | cat</pre>
</div>
</div>
<div class="paragraph">
<p>So we just check ourselves manually</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="kernel-module-utilities"><a class="anchor" href="#kernel-module-utilities"></a><a class="link" href="#kernel-module-utilities">12. Kernel module utilities</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="insmod"><a class="anchor" href="#insmod"></a><a class="link" href="#insmod">12.1. insmod</a></h3>
<div class="paragraph">
<p><a href="https://git.busybox.net/busybox/tree/modutils/insmod.c?h=1_29_3">Provided by BusyBox</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'insmod hello.ko'</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="myinsmod"><a class="anchor" href="#myinsmod"></a><a class="link" href="#myinsmod">12.2. myinsmod</a></h3>
<div class="paragraph">
<p>If you are feeling raw, you can insert and remove modules with our own minimal module inserter and remover!</p>
</div>
<div class="literalblock">
<div class="content">
<pre># init_module
./linux/myinsmod.out hello.ko
# finit_module
./linux/myinsmod.out hello.ko "" 1
./linux/myrmmod.out hello</pre>
</div>
</div>
<div class="paragraph">
<p>which teaches you how it is done from C code.</p>
</div>
<div class="paragraph">
<p>Source:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/myinsmod.c">userland/linux/myinsmod.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/myrmmod.c">userland/linux/myrmmod.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The Linux kernel offers two system calls for module insertion:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>init_module</code></p>
</li>
<li>
<p><code>finit_module</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>man init_module</pre>
</div>
</div>
<div class="paragraph">
<p>documents that:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The finit_module() system call is like init_module(), but reads the module to be loaded from the file descriptor fd. It is useful when the authenticity of a kernel module can be determined from its location in the filesystem; in cases where that is possible, the overhead of using cryptographically signed modules to determine the authenticity of a module can be avoided. The param_values argument is as for init_module().</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p><code>finit</code> is newer and was added only in v3.8. More rationale: <a href="https://lwn.net/Articles/519010/" class="bare">https://lwn.net/Articles/519010/</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/5947286/how-to-load-linux-kernel-modules-from-c-code" class="bare">https://stackoverflow.com/questions/5947286/how-to-load-linux-kernel-modules-from-c-code</a></p>
</div>
</div>
<div class="sect2">
<h3 id="modprobe"><a class="anchor" href="#modprobe"></a><a class="link" href="#modprobe">12.3. modprobe</a></h3>
<div class="paragraph">
<p>Implemented as a BusyBox applet by default: <a href="https://git.busybox.net/busybox/tree/modutils/modprobe.c?h=1_29_stable" class="bare">https://git.busybox.net/busybox/tree/modutils/modprobe.c?h=1_29_stable</a></p>
</div>
<div class="paragraph">
<p><code>modprobe</code> searches for modules installed under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls /lib/modules/&lt;kernel_version&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>and specified in the <code>modules.order</code> file.</p>
</div>
<div class="paragraph">
<p>This is the default install path for <code>CONFIG_SOME_MOD=m</code> modules built with <code>make modules_install</code> in the Linux kernel tree, with root path given by <code>INSTALL_MOD_PATH</code>, and therefore canonical in that sense.</p>
</div>
<div class="paragraph">
<p>Currently, there are only two kinds of kernel modules that you can try out with <code>modprobe</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>modules built with Buildroot, see: <a href="#kernel-modules-buildroot-package">Section 38.15.2.1, &#8220;kernel_modules buildroot package&#8221;</a></p>
</li>
<li>
<p>modules built from the kernel tree itself, see: <a href="#dummy-irq">Section 17.11.2, &#8220;dummy-irq&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We are not installing out custom <code>./build-modules</code> modules there, because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>we don&#8217;t know the right way. Why is there no <code>install</code> or <code>install_modules</code> target for kernel modules?</p>
<div class="paragraph">
<p>This can of course be solved by running Buildroot in verbose mode, and copying whatever it is doing, initial exploration at: <a href="https://stackoverflow.com/questions/22783793/how-to-install-kernel-modules-from-source-code-error-while-make-process/53169078#53169078" class="bare">https://stackoverflow.com/questions/22783793/how-to-install-kernel-modules-from-source-code-error-while-make-process/53169078#53169078</a></p>
</div>
</li>
<li>
<p>we would have to think how to not have to include the kernel modules twice in the root filesystem, but still have <a href="#9p">9P</a> working for fast development as described at: <a href="#your-first-kernel-module-hack">Section 2.2.2.2, &#8220;Your first kernel module hack&#8221;</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="kmod"><a class="anchor" href="#kmod"></a><a class="link" href="#kmod">12.4. kmod</a></h3>
<div class="paragraph">
<p>The more "reference" kernel.org implementation of <code>lsmod</code>, <code>insmod</code>, <code>rmmod</code>, etc.: <a href="https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git" class="bare">https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git</a></p>
</div>
<div class="paragraph">
<p>Default implementation on desktop distros such as Ubuntu 16.04, where e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls -l /bin/lsmod</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -&gt; kmod</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dpkg -l | grep -Ei</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ii  kmod                                        22-1ubuntu5                                         amd64        tools for managing Linux kernel modules</pre>
</div>
</div>
<div class="paragraph">
<p>BusyBox also implements its own version of those executables, see e.g. <a href="#modprobe">modprobe</a>. Here we will only describe features that differ from kmod to the BusyBox implementation.</p>
</div>
<div class="sect3">
<h4 id="module-init-tools"><a class="anchor" href="#module-init-tools"></a><a class="link" href="#module-init-tools">12.4.1. module-init-tools</a></h4>
<div class="paragraph">
<p>Name of a predecessor set of tools.</p>
</div>
</div>
<div class="sect3">
<h4 id="kmod-modprobe"><a class="anchor" href="#kmod-modprobe"></a><a class="link" href="#kmod-modprobe">12.4.2. kmod modprobe</a></h4>
<div class="paragraph">
<p>kmod&#8217;s <code>modprobe</code> can also load modules under different names to avoid conflicts, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo modprobe vmhgfs -o vm_hgfs</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="filesystems"><a class="anchor" href="#filesystems"></a><a class="link" href="#filesystems">13. Filesystems</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="overlayfs"><a class="anchor" href="#overlayfs"></a><a class="link" href="#overlayfs">13.1. OverlayFS</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/OverlayFS">OverlayFS</a> is a filesystem merged in the Linux kernel in 3.18.</p>
</div>
<div class="paragraph">
<p>As the name suggests, OverlayFS allows you to merge multiple directories into one. The following minimal runnable examples should give you an intuition on how it works:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/109413/how-do-i-use-overlayfs/1075564#1075564" class="bare">https://askubuntu.com/questions/109413/how-do-i-use-overlayfs/1075564#1075564</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/31044982/how-to-use-multiple-lower-layers-in-overlayfs/52792397#52792397" class="bare">https://stackoverflow.com/questions/31044982/how-to-use-multiple-lower-layers-in-overlayfs/52792397#52792397</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We are very interested in this filesystem because we are looking for a way to make host cross compiled executables appear on the guest root <code>/</code> without reboot.</p>
</div>
<div class="paragraph">
<p>This would have several advantages:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>makes it faster to test modified guest programs</p>
<div class="ulist">
<ul>
<li>
<p>not rebooting is fundamental for <a href="#gem5">gem5</a>, where the reboot is very costly.</p>
</li>
<li>
<p>no need to regenerate the root filesystem at all and reboot</p>
</li>
<li>
<p>overcomes the <code>check_bin_arch</code> problem as shown at: <a href="#rpath">Section 26.8, &#8220;Buildroot rebuild is slow when the root filesystem is large&#8221;</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>we could keep the base root filesystem very small, which implies:</p>
<div class="ulist">
<ul>
<li>
<p>less host disk usage, no need to copy the entire <code>./getvar out_rootfs_overlay_dir</code> to the image again</p>
</li>
<li>
<p>no need to worry about <a href="#br2-target-rootfs-ext2-size">BR2_TARGET_ROOTFS_EXT2_SIZE</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>We can already make host files appear on the guest with <a href="#9p">9P</a>, but they appear on a subdirectory instead of the root.</p>
</div>
<div class="paragraph">
<p>If they would appear on the root instead, that would be even more awesome, because you would just use the exact same paths relative to the root transparently.</p>
</div>
<div class="paragraph">
<p>For example, we wouldn&#8217;t have to mess around with variables such as <code>PATH</code> and <code>LD_LIBRARY_PATH</code>.</p>
</div>
<div class="paragraph">
<p>The idea is to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>9P mount our overlay directory <code>./getvar out_rootfs_overlay_dir</code> on the guest, which we already do at <code>/mnt/9p/out_rootfs_overlay</code></p>
</li>
<li>
<p>then create an overlay with that directory and the root, and <code>chroot</code> into it.</p>
<div class="paragraph">
<p>I was unable to mount directly to <code>/</code> avoid the <code>chroot</code>:
<strong> <a href="https://stackoverflow.com/questions/41119656/how-can-i-overlayfs-the-root-filesystem-on-linux" class="bare">https://stackoverflow.com/questions/41119656/how-can-i-overlayfs-the-root-filesystem-on-linux</a>
</strong> <a href="https://unix.stackexchange.com/questions/316018/how-to-use-overlayfs-to-protect-the-root-filesystem" class="bare">https://unix.stackexchange.com/questions/316018/how-to-use-overlayfs-to-protect-the-root-filesystem</a>
** <a href="https://unix.stackexchange.com/questions/420646/mount-root-as-overlayfs" class="bare">https://unix.stackexchange.com/questions/420646/mount-root-as-overlayfs</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>We already have a prototype of this running from <code>fstab</code> on guest at <code>/mnt/overlay</code>, but it has the following shortcomings:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>changes to underlying filesystems are not visible on the overlay unless you remount with <code>mount -r remount /mnt/overlay</code>, as mentioned <a href="https://github.com/torvalds/linux/blob/v4.18/Documentation/filesystems/overlayfs.txt#L332">on the kernel docs</a>:</p>
<div class="literalblock">
<div class="content">
<pre>Changes to the underlying filesystems while part of a mounted overlay
filesystem are not allowed.  If the underlying filesystem is changed,
the behavior of the overlay is undefined, though it will not result in
a crash or deadlock.</pre>
</div>
</div>
<div class="paragraph">
<p>This makes everything very inconvenient if you are inside <code>chroot</code> action. You would have to leave <code>chroot</code>, remount, then come back.</p>
</div>
</li>
<li>
<p>the overlay does not contain sub-filesystems, e.g. <code>/proc</code>. We would have to re-mount them. But should be doable with some automation.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Even more awesome than <code>chroot</code> would be to <code>pivot_root</code>, but I couldn&#8217;t get that working either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/28015688/pivot-root-device-or-resource-busy" class="bare">https://stackoverflow.com/questions/28015688/pivot-root-device-or-resource-busy</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/179788/pivot-root-device-or-resource-busy" class="bare">https://unix.stackexchange.com/questions/179788/pivot-root-device-or-resource-busy</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="secondary-disk"><a class="anchor" href="#secondary-disk"></a><a class="link" href="#secondary-disk">13.2. Secondary disk</a></h3>
<div class="paragraph">
<p>A simpler and possibly less overhead alternative to <a href="#9p">9P</a> would be to generate a secondary disk image with the benchmark you want to rebuild.</p>
</div>
<div class="paragraph">
<p>Then you can <code>umount</code> and re-mount on guest without reboot.</p>
</div>
<div class="paragraph">
<p>To build the secondary disk image run <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-disk2">build-disk2</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-disk2</pre>
</div>
</div>
<div class="paragraph">
<p>This will put the entire <a href="#out-rootfs-overlay-dir"><code>out_rootfs_overlay_dir</code></a> into a squashfs filesystem.</p>
</div>
<div class="paragraph">
<p>Then, if that filesystem is present, <code>./run</code> will automatically pass it as the second disk on the command line.</p>
</div>
<div class="paragraph">
<p>For example, from inside QEMU, you can mount that disk with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir /mnt/vdb
mount /dev/vdb /mnt/vdb
/mnt/vdb/lkmc/c/hello.out</pre>
</div>
</div>
<div class="paragraph">
<p>To update the secondary disk while a simulation is running to avoid rebooting, first unmount in the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>umount /mnt/vdb</pre>
</div>
</div>
<div class="paragraph">
<p>and then on the host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Edit the file.
vim userland/c/hello.c
./build-userland
./build-disk2</pre>
</div>
</div>
<div class="paragraph">
<p>and now you can re-run the updated version of the executable on the guest after remounting it.</p>
</div>
<div class="paragraph">
<p>gem5 fs.py support for multiple disks is discussed at: <a href="https://stackoverflow.com/questions/50862906/how-to-attach-multiple-disk-images-in-a-simulation-with-gem5-fs-py/51037661#51037661" class="bare">https://stackoverflow.com/questions/50862906/how-to-attach-multiple-disk-images-in-a-simulation-with-gem5-fs-py/51037661#51037661</a></p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="graphics"><a class="anchor" href="#graphics"></a><a class="link" href="#graphics">14. Graphics</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Both QEMU and gem5 are capable of outputting graphics to the screen, and taking mouse and keyboard input.</p>
</div>
<div class="paragraph">
<p><a href="https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux" class="bare">https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux</a></p>
</div>
<div class="sect2">
<h3 id="qemu-text-mode"><a class="anchor" href="#qemu-text-mode"></a><a class="link" href="#qemu-text-mode">14.1. QEMU text mode</a></h3>
<div class="paragraph">
<p>Text mode is the default mode for QEMU.</p>
</div>
<div class="paragraph">
<p>The opposite of text mode is <a href="#qemu-graphic-mode">QEMU graphic mode</a></p>
</div>
<div class="paragraph">
<p>In text mode, we just show the serial console directly on the current terminal, without opening a QEMU GUI window.</p>
</div>
<div class="paragraph">
<p>You cannot see any graphics from text mode, but text operations in this mode, including:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>scrolling up: <a href="#scroll-up-in-graphic-mode">Section 14.2.1, &#8220;Scroll up in graphic mode&#8221;</a></p>
</li>
<li>
<p>copy paste to and from the terminal</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>making this a good default, unless you really need to use with graphics.</p>
</div>
<div class="paragraph">
<p>Text mode works by sending the terminal character by character to a serial device.</p>
</div>
<div class="paragraph">
<p>This is different from a display screen, where each character is a bunch of pixels, and it would be much harder to convert that into actual terminal text.</p>
</div>
<div class="paragraph">
<p>For more details, see:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux" class="bare">https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux</a></p>
</li>
<li>
<p><a href="#tty">TTY</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Note that you can still see an image even in text mode with the VNC:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --vnc</pre>
</div>
</div>
<div class="paragraph">
<p>and on another terminal:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./vnc</pre>
</div>
</div>
<div class="paragraph">
<p>but there is not terminal on the VNC window, just the <a href="#config-logo">CONFIG_LOGO</a> penguin.</p>
</div>
<div class="sect3">
<h4 id="quit-qemu-from-text-mode"><a class="anchor" href="#quit-qemu-from-text-mode"></a><a class="link" href="#quit-qemu-from-text-mode">14.1.1. Quit QEMU from text mode</a></h4>
<div class="paragraph">
<p><a href="https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui" class="bare">https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui</a></p>
</div>
<div class="paragraph">
<p>However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.</p>
</div>
<div class="paragraph">
<p>The simplest way to quit QEMU, is to do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-A X</pre>
</div>
</div>
<div class="paragraph">
<p>Alternative methods include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>quit</code> command on the <a href="#qemu-monitor">QEMU monitor</a></p>
</li>
<li>
<p><code>pkill qemu</code></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qemu-graphic-mode"><a class="anchor" href="#qemu-graphic-mode"></a><a class="link" href="#qemu-graphic-mode">14.2. QEMU graphic mode</a></h3>
<div class="paragraph">
<p>Enable graphic mode with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: you see a penguin due to <a href="#config-logo">CONFIG_LOGO</a>.</p>
</div>
<div class="paragraph">
<p>For a more exciting GUI experience, see: <a href="#x11">Section 14.4, &#8220;X11 Buildroot&#8221;</a></p>
</div>
<div class="paragraph">
<p>Text mode is the default due to the following considerable advantages:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>copy and paste commands and stdout output to / from host</p>
</li>
<li>
<p>get full panic traces when you start making the kernel crash :-) See also: <a href="https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic" class="bare">https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic</a></p>
</li>
<li>
<p>have a large scroll buffer, and be able to search it, e.g. by using tmux on host</p>
</li>
<li>
<p>one less window floating around to think about in addition to your shell :-)</p>
</li>
<li>
<p>graphics mode has only been properly tested on <code>x86_64</code>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Text mode has the following limitations over graphics mode:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>you can&#8217;t see graphics such as those produced by <a href="#x11">X11 Buildroot</a></p>
</li>
<li>
<p>very early kernel messages such as <code>early console in extract_kernel</code> only show on the GUI, since at such early stages, not even the serial has been setup.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>x86_64</code> has a VGA device enabled by default, as can be seen as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor info qtree</pre>
</div>
</div>
<div class="paragraph">
<p>and the Linux kernel picks it up through the <a href="https://en.wikipedia.org/wiki/Linux_framebuffer">fbdev</a> graphics system as can be seen from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /dev/urandom &gt; /dev/fb0</pre>
</div>
</div>
<div class="paragraph">
<p>flooding the screen with colors. See also: <a href="https://superuser.com/questions/223094/how-do-i-know-if-i-have-kms-enabled" class="bare">https://superuser.com/questions/223094/how-do-i-know-if-i-have-kms-enabled</a></p>
</div>
<div class="sect3">
<h4 id="scroll-up-in-graphic-mode"><a class="anchor" href="#scroll-up-in-graphic-mode"></a><a class="link" href="#scroll-up-in-graphic-mode">14.2.1. Scroll up in graphic mode</a></h4>
<div class="paragraph">
<p>Scroll up in <a href="#qemu-graphic-mode">QEMU graphic mode</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Shift-PgUp</pre>
</div>
</div>
<div class="paragraph">
<p>but I never managed to increase that buffer:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition" class="bare">https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty" class="bare">https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The superior alternative is to use text mode and GNU screen or <a href="#tmux">tmux</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-graphic-mode-arm"><a class="anchor" href="#qemu-graphic-mode-arm"></a><a class="link" href="#qemu-graphic-mode-arm">14.2.2. QEMU Graphic mode arm</a></h4>
<div class="sect4">
<h5 id="qemu-graphic-mode-arm-terminal"><a class="anchor" href="#qemu-graphic-mode-arm-terminal"></a><a class="link" href="#qemu-graphic-mode-arm-terminal">14.2.2.1. QEMU graphic mode arm terminal</a></h5>
<div class="paragraph">
<p>TODO: on arm, we see the penguin and some boot messages, but don&#8217;t get a shell at then end:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>I think it does not work because the graphic window is <a href="#drm">DRM</a> only, i.e.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /dev/urandom &gt; /dev/fb0</pre>
</div>
</div>
<div class="paragraph">
<p>fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat: write error: No space left on device</pre>
</div>
</div>
<div class="paragraph">
<p>and has no effect, and the Linux kernel does not appear to have a built-in DRM console as it does for fbdev with <a href="#fbcon">fbcon</a>.</p>
</div>
<div class="paragraph">
<p>There is however one out-of-tree implementation: <a href="#kmscon">kmscon</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="qemu-graphic-mode-arm-terminal-implementation"><a class="anchor" href="#qemu-graphic-mode-arm-terminal-implementation"></a><a class="link" href="#qemu-graphic-mode-arm-terminal-implementation">14.2.2.2. QEMU graphic mode arm terminal implementation</a></h5>
<div class="paragraph">
<p><code>arm</code> and <code>aarch64</code> rely on the QEMU CLI option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-device virtio-gpu-pci</pre>
</div>
</div>
<div class="paragraph">
<p>and the kernel config options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y</pre>
</div>
</div>
<div class="paragraph">
<p>Unlike x86, <code>arm</code> and <code>aarch64</code> don&#8217;t have a display device attached by default, thus the need for <code>virtio-gpu-pci</code>.</p>
</div>
<div class="paragraph">
<p>See also <a href="https://wiki.qemu.org/Documentation/Platforms/ARM" class="bare">https://wiki.qemu.org/Documentation/Platforms/ARM</a> (recently edited and corrected by yours truly&#8230;&#8203; :-)).</p>
</div>
</div>
<div class="sect4">
<h5 id="qemu-graphic-mode-arm-vga"><a class="anchor" href="#qemu-graphic-mode-arm-vga"></a><a class="link" href="#qemu-graphic-mode-arm-vga">14.2.2.3. QEMU graphic mode arm VGA</a></h5>
<div class="paragraph">
<p>TODO: how to use VGA on ARM? <a href="https://stackoverflow.com/questions/20811203/how-can-i-output-to-vga-through-qemu-arm" class="bare">https://stackoverflow.com/questions/20811203/how-can-i-output-to-vga-through-qemu-arm</a> Tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-device VGA</pre>
</div>
</div>
<div class="paragraph">
<p>But <a href="https://github.com/qemu/qemu/blob/v2.12.0/docs/config/mach-virt-graphical.cfg#L264" class="bare">https://github.com/qemu/qemu/blob/v2.12.0/docs/config/mach-virt-graphical.cfg#L264</a> says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># We use virtio-gpu because the legacy VGA framebuffer is
# very troublesome on aarch64, and virtio-gpu is the only
# video device that doesn't implement it.</pre>
</div>
</div>
<div class="paragraph">
<p>so maybe it is not possible?</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-graphic-mode"><a class="anchor" href="#gem5-graphic-mode"></a><a class="link" href="#gem5-graphic-mode">14.3. gem5 graphic mode</a></h3>
<div class="paragraph">
<p>gem5 does not have a "text mode", since it cannot redirect the Linux terminal to same host terminal where the executable is running: you are always forced to connect to the terminal with <code>gem-shell</code>.</p>
</div>
<div class="paragraph">
<p>TODO could not get it working on <code>x86_64</code>, only ARM.</p>
</div>
<div class="paragraph">
<p>Overview: <a href="https://stackoverflow.com/questions/50364863/how-to-get-graphical-gui-output-and-user-touch-keyboard-mouse-input-in-a-ful/50364864#50364864" class="bare">https://stackoverflow.com/questions/50364863/how-to-get-graphical-gui-output-and-user-touch-keyboard-mouse-input-in-a-ful/50364864#50364864</a></p>
</div>
<div class="paragraph">
<p>More concretely, first build the kernel with the <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a>, and then run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --arch arm \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
./run --arch arm --emulator gem5 --linux-build-id gem5-v4.15</pre>
</div>
</div>
<div class="paragraph">
<p>and then on another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vinagre localhost:5900</pre>
</div>
</div>
<div class="paragraph">
<p>The <a href="#config-logo">CONFIG_LOGO</a> penguin only appears after several seconds, together with kernel messages of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    0.152755] [drm] found ARM HDLCD version r0p0
[    0.152790] hdlcd 2b000000.hdlcd: bound virt-encoder (ops 0x80935f94)
[    0.152795] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.152799] [drm] No driver support for vblank timestamp query.
[    0.215179] Console: switching to colour frame buffer device 240x67
[    0.230389] hdlcd 2b000000.hdlcd: fb0:  frame buffer device
[    0.230509] [drm] Initialized hdlcd 1.0.0 20151021 for 2b000000.hdlcd on minor 0</pre>
</div>
</div>
<div class="paragraph">
<p>The port <code>5900</code> is incremented by one if you already have something running on that port, <code>gem5</code> stdout tells us the right port on stdout as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system.vncserver: Listening for connections on port 5900</pre>
</div>
</div>
<div class="paragraph">
<p>and when we connect it shows a message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: VNC client attached</pre>
</div>
</div>
<div class="paragraph">
<p>Alternatively, you can also dump each new frame to an image file with <code>--frame-capture</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch arm \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
  -- --frame-capture \
;</pre>
</div>
</div>
<div class="paragraph">
<p>This creates on compressed PNG whenever the screen image changes inside the <a href="#m5out-directory">m5out directory</a> with filename of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>frames_system.vncserver/fb.&lt;frame-index&gt;.&lt;timestamp&gt;.png.gz</pre>
</div>
</div>
<div class="paragraph">
<p>It is fun to see how we get one new frame whenever the white underscore cursor appears and reappears under the penguin!</p>
</div>
<div class="paragraph">
<p>The last frame is always available uncompressed at: <code>system.framebuffer.png</code>.</p>
</div>
<div class="paragraph">
<p>TODO <a href="#kmscube">kmscube</a> failed on <code>aarch64</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>kmscube[706]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006, in libgbm.so.1.0.0[7fbf6a6000+e000]</pre>
</div>
</div>
<div class="paragraph">
<p>Tested on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/38fd6153d965ba20145f53dc1bb3ba34b336bde9">38fd6153d965ba20145f53dc1bb3ba34b336bde9</a></p>
</div>
<div class="sect3">
<h4 id="graphic-mode-gem5-aarch64"><a class="anchor" href="#graphic-mode-gem5-aarch64"></a><a class="link" href="#graphic-mode-gem5-aarch64">14.3.1. Graphic mode gem5 aarch64</a></h4>
<div class="paragraph">
<p>For <code>aarch64</code> we also need to configure the kernel with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/display">linux_config/display</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git -C "$(./getvar linux_source_dir)" fetch https://gem5.googlesource.com/arm/linux gem5/v4.15:gem5/v4.15
git -C "$(./getvar linux_source_dir)" checkout gem5/v4.15
./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/display \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
git -C "$(./getvar linux_source_dir)" checkout -
./run --arch aarch64 --emulator gem5 --linux-build-id gem5-v4.15</pre>
</div>
</div>
<div class="paragraph">
<p>This is because the gem5 <code>aarch64</code> defconfig does not enable HDLCD like the 32 bit one <code>arm</code> one for some reason.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-graphic-mode-dp650"><a class="anchor" href="#gem5-graphic-mode-dp650"></a><a class="link" href="#gem5-graphic-mode-dp650">14.3.2. gem5 graphic mode DP650</a></h4>
<div class="paragraph">
<p>TODO get working. There is an unmerged patchset at: <a href="https://gem5-review.googlesource.com/c/public/gem5/+/11036/1" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/11036/1</a></p>
</div>
<div class="paragraph">
<p>The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and <a href="https://github.com/torvalds/linux/blob/v4.19/drivers/gpu/drm/arm/Kconfig#L39">in-tree Linux kernel support</a>, that information cannot be secret?</p>
</div>
<div class="paragraph">
<p>The key option to enable support in Linux is <code>DRM_MALI_DISPLAY=y</code> which we enable at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/display">linux_config/display</a>.</p>
</div>
<div class="paragraph">
<p>Build the kernel exactly as for <a href="#graphic-mode-gem5-aarch64">Graphic mode gem5 aarch64</a> and then run with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --dp650 --emulator gem5 --linux-build-id gem5-v4.15</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-graphic-mode-internals"><a class="anchor" href="#gem5-graphic-mode-internals"></a><a class="link" href="#gem5-graphic-mode-internals">14.3.3. gem5 graphic mode internals</a></h4>
<div class="paragraph">
<p>We cannot use mainline Linux because the <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a> are required at least to provide the <code>CONFIG_DRM_VIRT_ENCODER</code> option.</p>
</div>
<div class="paragraph">
<p>gem5 emulates the <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0541c/CHDBAIDI.html">HDLCD</a> ARM Holdings hardware for <code>arm</code> and <code>aarch64</code>.</p>
</div>
<div class="paragraph">
<p>The kernel uses HDLCD to implement the <a href="#drm">DRM</a> interface, the required kernel config options are present at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/display">linux_config/display</a>.</p>
</div>
<div class="paragraph">
<p>TODO: minimize out the <code>--custom-config-file</code>. If we just remove it on <code>arm</code>: it does not work with a failing dmesg:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    0.066208] [drm] found ARM HDLCD version r0p0
[    0.066241] hdlcd 2b000000.hdlcd: bound virt-encoder (ops drm_vencoder_ops)
[    0.066247] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.066252] [drm] No driver support for vblank timestamp query.
[    0.066276] hdlcd 2b000000.hdlcd: Cannot do DMA to address 0x0000000000000000
[    0.066281] swiotlb: coherent allocation failed for device 2b000000.hdlcd size=8294400
[    0.066288] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0 #1
[    0.066293] Hardware name: V2P-AARCH64 (DT)
[    0.066296] Call trace:
[    0.066301]  dump_backtrace+0x0/0x1b0
[    0.066306]  show_stack+0x24/0x30
[    0.066311]  dump_stack+0xb8/0xf0
[    0.066316]  swiotlb_alloc_coherent+0x17c/0x190
[    0.066321]  __dma_alloc+0x68/0x160
[    0.066325]  drm_gem_cma_create+0x98/0x120
[    0.066330]  drm_fbdev_cma_create+0x74/0x2e0
[    0.066335]  __drm_fb_helper_initial_config_and_unlock+0x1d8/0x3a0
[    0.066341]  drm_fb_helper_initial_config+0x4c/0x58
[    0.066347]  drm_fbdev_cma_init_with_funcs+0x98/0x148
[    0.066352]  drm_fbdev_cma_init+0x40/0x50
[    0.066357]  hdlcd_drm_bind+0x220/0x428
[    0.066362]  try_to_bring_up_master+0x21c/0x2b8
[    0.066367]  component_master_add_with_match+0xa8/0xf0
[    0.066372]  hdlcd_probe+0x60/0x78
[    0.066377]  platform_drv_probe+0x60/0xc8
[    0.066382]  driver_probe_device+0x30c/0x478
[    0.066388]  __driver_attach+0x10c/0x128
[    0.066393]  bus_for_each_dev+0x70/0xb0
[    0.066398]  driver_attach+0x30/0x40
[    0.066402]  bus_add_driver+0x1d0/0x298
[    0.066408]  driver_register+0x68/0x100
[    0.066413]  __platform_driver_register+0x54/0x60
[    0.066418]  hdlcd_platform_driver_init+0x20/0x28
[    0.066424]  do_one_initcall+0x44/0x130
[    0.066428]  kernel_init_freeable+0x13c/0x1d8
[    0.066433]  kernel_init+0x18/0x108
[    0.066438]  ret_from_fork+0x10/0x1c
[    0.066444] hdlcd 2b000000.hdlcd: Failed to set initial hw configuration.
[    0.066470] hdlcd 2b000000.hdlcd: master bind failed: -12
[    0.066477] hdlcd: probe of 2b000000.hdlcd failed with error -12</pre>
</div>
</div>
<div class="paragraph">
<p>So what other options are missing from <code>gem5_defconfig</code>? It would be cool to minimize it out to better understand the options.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x11"><a class="anchor" href="#x11"></a><a class="link" href="#x11">14.4. X11 Buildroot</a></h3>
<div class="paragraph">
<p>Once you&#8217;ve seen the <code>CONFIG_LOGO</code> penguin as a sanity check, you can try to go for a cooler X11 Buildroot setup.</p>
</div>
<div class="paragraph">
<p>Build and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config-fragment buildroot_config/x11
./run --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Inside QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>startx</pre>
</div>
</div>
<div class="paragraph">
<p>And then from the GUI you can start exciting graphical programs such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>xcalc
xeyes</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: <a href="#image-x11">Figure 1, &#8220;X11 Buildroot graphical user interface screenshot&#8221;</a></p>
</div>
<div id="image-x11" class="imageblock">
<div class="content">
<a class="image" href="x11.png"><img src="x11.png" alt="x11"></a>
</div>
<div class="title">Figure 1. X11 Buildroot graphical user interface screenshot</div>
</div>
<div class="paragraph">
<p>We don&#8217;t build X11 by default because it takes a considerable amount of time (about 20%), and is not expected to be used by most users: you need to pass the <code>-x</code> flag to enable it.</p>
</div>
<div class="paragraph">
<p>More details: <a href="https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116" class="bare">https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116</a></p>
</div>
<div class="paragraph">
<p>Not sure how well that graphics stack represents real systems, but if it does it would be a good way to understand how it works.</p>
</div>
<div class="paragraph">
<p>To x11 packages have an <code>xserver</code> prefix as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config-fragment buildroot_config/x11 -- xserver_xorg-server-reconfigure</pre>
</div>
</div>
<div class="paragraph">
<p>the easiest way to find them out is to just list <code>"$(./getvar buildroot_build_build_dir)/x*</code>.</p>
</div>
<div class="paragraph">
<p>TODO as of: c2696c978d6ca88e8b8599c92b1beeda80eb62b2 I noticed that <code>startx</code> leads to a <a href="#bug-on">BUG_ON</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    2.809104] WARNING: CPU: 0 PID: 51 at drivers/gpu/drm/ttm/ttm_bo_vm.c:304 ttm_bo_vm_open+0x37/0x40</pre>
</div>
</div>
<div class="sect3">
<h4 id="x11-buildroot-mouse-not-moving"><a class="anchor" href="#x11-buildroot-mouse-not-moving"></a><a class="link" href="#x11-buildroot-mouse-not-moving">14.4.1. X11 Buildroot mouse not moving</a></h4>
<div class="paragraph">
<p>TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I&#8217;m 100% sure that it was working before, but I didn&#8217;t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu" class="bare">https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator" class="bare">https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>-show-cursor</code> did not help, I just get to see the host cursor, but the guest cursor still does not move.</p>
</div>
<div class="paragraph">
<p>Doing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>watch -n 1 grep i8042 /proc/interrupts</pre>
</div>
</div>
<div class="paragraph">
<p>shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>QEMU. Same behaviour if I try the host&#8217;s QEMU 2.10.1 however.</p>
</li>
<li>
<p>X11 configuration. We do have <code>BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y</code>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>/var/log/Xorg.0.log</code> contains the following interesting lines:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[    27.549] (II) LoadModule: "mouse"
[    27.549] (II) Loading /usr/lib/xorg/modules/input/mouse_drv.so
[    27.590] (EE) &lt;default pointer&gt;: Cannot find which device to use.
[    27.590] (EE) &lt;default pointer&gt;: cannot open input device
[    27.590] (EE) PreInit returned 2 for "&lt;default pointer&gt;"
[    27.590] (II) UnloadModule: "mouse"</pre>
</div>
</div>
<div class="paragraph">
<p>The file <code>/dev/inputs/mice</code> does not exist.</p>
</div>
<div class="paragraph">
<p>Note that our current link:kernel_confi_fragment sets:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set</pre>
</div>
</div>
<div class="paragraph">
<p>for gem5, so you might want to remove those lines to debug this.</p>
</div>
</div>
<div class="sect3">
<h4 id="x11-buildroot-arm"><a class="anchor" href="#x11-buildroot-arm"></a><a class="link" href="#x11-buildroot-arm">14.4.2. X11 Buildroot ARM</a></h4>
<div class="paragraph">
<p>On ARM, <code>startx</code> hangs at a message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vgaarb: this pci device is not a vga device</pre>
</div>
</div>
<div class="paragraph">
<p>and nothing shows on the screen, and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep EE /var/log/Xorg.0.log</pre>
</div>
</div>
<div class="paragraph">
<p>says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>(EE) Failed to load module "modesetting" (module does not exist, 0)</pre>
</div>
</div>
<div class="paragraph">
<p>A friend told me this but I haven&#8217;t tried it yet:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>xf86-video-modesetting</code> is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.</p>
</li>
<li>
<p><code>xf86-video-fbdev</code> should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the <code>Xorg.conf</code></p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="networking"><a class="anchor" href="#networking"></a><a class="link" href="#networking">15. Networking</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="enable-networking"><a class="anchor" href="#enable-networking"></a><a class="link" href="#enable-networking">15.1. Enable networking</a></h3>
<div class="paragraph">
<p>We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: <a href="#resource-tradeoff-guidelines">Section 38.20.3, &#8220;Resource tradeoff guidelines&#8221;</a></p>
</div>
<div class="paragraph">
<p>To enable networking on Buildroot, simply run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ifup -a</pre>
</div>
</div>
<div class="paragraph">
<p>That command goes over all (<code>-a</code>) the interfaces in <code>/etc/network/interfaces</code> and brings them up.</p>
</div>
<div class="paragraph">
<p>Then test it with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>wget google.com
cat index.html</pre>
</div>
</div>
<div class="paragraph">
<p>Disable networking with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ifdown -a</pre>
</div>
</div>
<div class="paragraph">
<p>To enable networking by default after boot, use the methods documented at <a href="#init-busybox">Run command at the end of BusyBox init</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="ping"><a class="anchor" href="#ping"></a><a class="link" href="#ping">15.2. ping</a></h3>
<div class="paragraph">
<p><code>ping</code> does not work within QEMU by default, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ping google.com</pre>
</div>
</div>
<div class="paragraph">
<p>hangs after printing the header:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>PING google.com (216.58.204.46): 56 data bytes</pre>
</div>
</div>
<div class="paragraph">
<p>Here Ciro describes how to get it working: <a href="https://unix.stackexchange.com/questions/473448/how-to-ping-from-the-qemu-guest-to-an-external-url" class="bare">https://unix.stackexchange.com/questions/473448/how-to-ping-from-the-qemu-guest-to-an-external-url</a></p>
</div>
<div class="paragraph">
<p>Further bibliography: <a href="https://superuser.com/questions/787400/qemu-user-mode-networking-doesnt-work" class="bare">https://superuser.com/questions/787400/qemu-user-mode-networking-doesnt-work</a></p>
</div>
</div>
<div class="sect2">
<h3 id="guest-host-networking"><a class="anchor" href="#guest-host-networking"></a><a class="link" href="#guest-host-networking">15.3. Guest host networking</a></h3>
<div class="paragraph">
<p>In this section we discuss how to interact between the guest and the host through networking.</p>
</div>
<div class="paragraph">
<p>First ensure that you can access the external network since that is easier to get working, see: <a href="#networking">Section 15, &#8220;Networking&#8221;</a>.</p>
</div>
<div class="sect3">
<h4 id="host-to-guest-networking"><a class="anchor" href="#host-to-guest-networking"></a><a class="link" href="#host-to-guest-networking">15.3.1. Host to guest networking</a></h4>
<div class="sect4">
<h5 id="nc-host-to-guest"><a class="anchor" href="#nc-host-to-guest"></a><a class="link" href="#nc-host-to-guest">15.3.1.1. nc host to guest</a></h5>
<div class="paragraph">
<p>With <code>nc</code> we can create the most minimal example possible as a sanity check.</p>
</div>
<div class="paragraph">
<p>On guest run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>nc -l -p 45455</pre>
</div>
</div>
<div class="paragraph">
<p>Then on host run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo asdf | nc localhost 45455</pre>
</div>
</div>
<div class="paragraph">
<p><code>asdf</code> appears on the guest.</p>
</div>
<div class="paragraph">
<p>This uses:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>BusyBox' <code>nc</code> utility, which is enabled with <code>CONFIG_NC=y</code></p>
</li>
<li>
<p><code>nc</code> from the <code>netcat-openbsd</code> package on an Ubuntu 18.04 host</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Only this specific port works by default since we have forwarded it on the QEMU command line.</p>
</div>
<div class="paragraph">
<p>We us this exact procedure to connect to <a href="#gdbserver">gdbserver</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="ssh-into-guest"><a class="anchor" href="#ssh-into-guest"></a><a class="link" href="#ssh-into-guest">15.3.1.2. ssh into guest</a></h5>
<div class="paragraph">
<p>Not enabled by default due to the build / runtime overhead. To enable, build with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_OPENSSH=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Then inside the guest turn on sshd:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./sshd.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/sshd.sh">rootfs_overlay/lkmc/sshd.sh</a></p>
</div>
<div class="paragraph">
<p>And finally on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ssh root@localhost -p 45456</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://unix.stackexchange.com/questions/124681/how-to-ssh-from-host-to-guest-using-qemu/307557#307557" class="bare">https://unix.stackexchange.com/questions/124681/how-to-ssh-from-host-to-guest-using-qemu/307557#307557</a></p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-host-to-guest-networking"><a class="anchor" href="#gem5-host-to-guest-networking"></a><a class="link" href="#gem5-host-to-guest-networking">15.3.1.3. gem5 host to guest networking</a></h5>
<div class="paragraph">
<p>Could not do port forwarding from host to guest, and therefore could not use <code>gdbserver</code>: <a href="https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5" class="bare">https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="guest-to-host-networking"><a class="anchor" href="#guest-to-host-networking"></a><a class="link" href="#guest-to-host-networking">15.3.2. Guest to host networking</a></h4>
<div class="paragraph">
<p>First <a href="#enable-networking">Enable networking</a>.</p>
</div>
<div class="paragraph">
<p>Then in the host, start a server:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>python -m SimpleHTTPServer 8000</pre>
</div>
</div>
<div class="paragraph">
<p>And then in the guest, find the IP we need to hit with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ip rounte</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 scope link  src 10.0.2.15</pre>
</div>
</div>
<div class="paragraph">
<p>so we use in the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>wget 10.0.2.2:8000</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835" class="bare">https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/78953/qemu-how-to-ping-host-network/547698#547698" class="bare">https://unix.stackexchange.com/questions/78953/qemu-how-to-ping-host-network/547698#547698</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="9p"><a class="anchor" href="#9p"></a><a class="link" href="#9p">15.4. 9P</a></h3>
<div class="paragraph">
<p>The <a href="https://en.wikipedia.org/wiki/9P_(protocol)">9p protocol</a> allows the guest to mount a host directory.</p>
</div>
<div class="paragraph">
<p>Both QEMU and <a href="#gem5-9p">gem5 9P</a> support 9P.</p>
</div>
<div class="sect3">
<h4 id="9p-vs-nfs"><a class="anchor" href="#9p-vs-nfs"></a><a class="link" href="#9p-vs-nfs">15.4.1. 9P vs NFS</a></h4>
<div class="paragraph">
<p>All of 9P and NFS (and sshfs) allow sharing directories between guest and host.</p>
</div>
<div class="paragraph">
<p>Advantages of 9P</p>
</div>
<div class="ulist">
<ul>
<li>
<p>requires <code>sudo</code> on the host to mount</p>
</li>
<li>
<p>we could share a guest directory to the host, but this would require running a server on the guest, which adds <a href="#resource-tradeoff-guidelines">simulation overhead</a></p>
<div class="paragraph">
<p>Furthermore, this would be inconvenient, since what we usually want to do is to share host cross built files with the guest, and to do that we would have to copy the files over after the guest starts the server.</p>
</div>
</li>
<li>
<p>QEMU implements 9P natively, which makes it very stable and convenient, and must mean it is a simpler protocol than NFS as one would expect.</p>
<div class="paragraph">
<p>This is not the case for gem5 7bfb7f3a43f382eb49853f47b140bfd6caad0fb8 unfortunately, which relies on the <a href="https://github.com/chaos/diod">diod</a> host daemon, although it is not unfeasible that future versions could implement it natively as well.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Advantages of NFS:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>way more widely used and therefore stable and available, not to mention that it also works on real hardware.</p>
</li>
<li>
<p>the name does not start with a digit, which is an invalid identifier in all programming languages known to man. Who in their right mind would call a software project as such? It does not even match the natural order of Plan 9; Plan then 9: P9!</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="9p-getting-started"><a class="anchor" href="#9p-getting-started"></a><a class="link" href="#9p-getting-started">15.4.2. 9P getting started</a></h4>
<div class="paragraph">
<p>As usual, we have already set everything up for you. On host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd "$(./getvar p9_dir)"
uname -a &gt; host</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd /mnt/9p/data
cat host
uname -a &gt; guest</pre>
</div>
</div>
<div class="paragraph">
<p>Host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat guest</pre>
</div>
</div>
<div class="paragraph">
<p>The main ingredients for this are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>9P</code> settings in our <a href="#kernel-configs-about">kernel configs</a></p>
</li>
<li>
<p><code>9p</code> entry on our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/fstab">rootfs_overlay/etc/fstab</a></p>
<div class="paragraph">
<p>Alternatively, you could also mount your own with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir /mnt/my9p
mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p</pre>
</div>
</div>
<div class="paragraph">
<p>where mount tag <code>host0</code> is set by the emulator (<code>mount_tag</code> flag on QEMU CLI), and can be found in the guest with: <code>cat /sys/bus/virtio/drivers/9pnet_virtio/virtio0/mount_tag</code> as documented at: <a href="https://www.kernel.org/doc/Documentation/filesystems/9p.txt" class="bare">https://www.kernel.org/doc/Documentation/filesystems/9p.txt</a>.</p>
</div>
</li>
<li>
<p>Launch QEMU with <code>-virtfs</code> as in your <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> script</p>
<div class="paragraph">
<p>When we tried:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>security_model=mapped</pre>
</div>
</div>
<div class="paragraph">
<p>writes from guest failed due to user mismatch problems: <a href="https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu" class="bare">https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu" class="bare">https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu</a></p>
</li>
<li>
<p><a href="https://wiki.qemu.org/Documentation/9psetup" class="bare">https://wiki.qemu.org/Documentation/9psetup</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gem5-9p"><a class="anchor" href="#gem5-9p"></a><a class="link" href="#gem5-9p">15.4.3. gem5 9P</a></h4>
<div class="paragraph">
<p>Is possible on aarch64 as shown at: <a href="https://gem5-review.googlesource.com/c/public/gem5/+/22831" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/22831</a>, and it is just a matter of exposing to X86 for those that want it.</p>
</div>
<div class="paragraph">
<p>Enable it by passing the <code>--vio-9p</code> option on the fs.py gem5 command line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 -- --vio-9p</pre>
</div>
</div>
<div class="paragraph">
<p>Then on the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir -p /mnt/9p/gem5
mount -t 9p -o trans=virtio,version=9p2000.L,aname=/path/to/linux-kernel-module-cheat/out/run/gem5/aarch64/0/m5out/9p/share gem5 /mnt/9p/gem5
echo asdf &gt; /mnt/9p/gem5/qwer</pre>
</div>
</div>
<div class="paragraph">
<p>Yes, you have to pass the full path to the directory on the host. Yes, this is horrible.</p>
</div>
<div class="paragraph">
<p>The shared directory is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>out/run/gem5/aarch64/0/m5out/9p/share</pre>
</div>
</div>
<div class="paragraph">
<p>so we can observe the file the guest wrote from the host with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>out/run/gem5/aarch64/0/m5out/9p/share/qwer</pre>
</div>
</div>
<div class="paragraph">
<p>and vice versa:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo zxvc &gt; out/run/gem5/aarch64/0/m5out/9p/share/qwer</pre>
</div>
</div>
<div class="paragraph">
<p>is now visible from the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /mnt/9p/gem5/qwer</pre>
</div>
</div>
<div class="paragraph">
<p>Checkpoint restore with an open mount will likely fail because gem5 uses an ugly external executable to implement diod. The protocol is not very complex, and QEMU implements it in-tree, which is what gem5 should do as well at some point.</p>
</div>
<div class="paragraph">
<p>Also checkpoint without <code>--vio-9p</code> and restore with <code>--vio-9p</code> did not work either, the mount fails.</p>
</div>
<div class="paragraph">
<p>However, this did work, on guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>unmount /mnt/9p/gem5
m5 checkpoint</pre>
</div>
</div>
<div class="paragraph">
<p>then restore with the detalied CPU of interest e.g.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 -- --vio-9p --cpu-type DerivO3CPU --caches</pre>
</div>
</div>
<div class="paragraph">
<p>Tested on gem5 b2847f43c91e27f43bd4ac08abd528efcf00f2fd, LKMC 52a5fdd7c1d6eadc5900fc76e128995d4849aada.</p>
</div>
</div>
<div class="sect3">
<h4 id="nfs"><a class="anchor" href="#nfs"></a><a class="link" href="#nfs">15.4.4. NFS</a></h4>
<div class="paragraph">
<p>TODO: get working.</p>
</div>
<div class="paragraph">
<p><a href="#9p">9P</a> is better with emulation, but let&#8217;s just get this working for fun.</p>
</div>
<div class="paragraph">
<p>First make sure that this works: <a href="#guest-to-host-networking">Section 15.3.2, &#8220;Guest to host networking&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Then, build the kernel with NFS support:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --config-fragment linux_config/nfs</pre>
</div>
</div>
<div class="paragraph">
<p>Now on host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install nfs-kernel-server</pre>
</div>
</div>
<div class="paragraph">
<p>Now edit <code>/etc/exports</code> to contain:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/tmp *(rw,sync,no_root_squash,no_subtree_check)</pre>
</div>
</div>
<div class="paragraph">
<p>and restart the server:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo systemctl restart nfs-kernel-server</pre>
</div>
</div>
<div class="paragraph">
<p>Now on guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir /mnt/nfs
mount -t nfs 10.0.2.2:/tmp /mnt/nfs</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: failing with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mount: mounting 10.0.2.2:/tmp on /mnt/nfs failed: No such device</pre>
</div>
</div>
<div class="paragraph">
<p>And now the <code>/tmp</code> directory from host is not mounted on guest!</p>
</div>
<div class="paragraph">
<p>If you don&#8217;t want to start the NFS server after the next boot automatically so save resources, <a href="https://askubuntu.com/questions/19320/how-to-enable-or-disable-services">do</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>systemctl disable nfs-kernel-server</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="operating-systems"><a class="anchor" href="#operating-systems"></a><a class="link" href="#operating-systems">16. Operating systems</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Operating_system" class="bare">https://en.wikipedia.org/wiki/Operating_system</a></p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#linux-kernel">Linux kernel</a></p>
</li>
<li>
<p><a href="#freebsd">FreeBSD</a></p>
</li>
<li>
<p><a href="#rtos">RTOS</a></p>
</li>
<li>
<p><a href="#xen">Xen</a></p>
</li>
<li>
<p><a href="#u-boot">U-Boot</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="linux-kernel"><a class="anchor" href="#linux-kernel"></a><a class="link" href="#linux-kernel">17. Linux kernel</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Linux_kernel" class="bare">https://en.wikipedia.org/wiki/Linux_kernel</a></p>
</div>
<div class="sect2">
<h3 id="linux-kernel-configuration"><a class="anchor" href="#linux-kernel-configuration"></a><a class="link" href="#linux-kernel-configuration">17.1. Linux kernel configuration</a></h3>
<div class="sect3">
<h4 id="modify-kernel-config"><a class="anchor" href="#modify-kernel-config"></a><a class="link" href="#modify-kernel-config">17.1.1. Modify kernel config</a></h4>
<div class="paragraph">
<p>To modify a single option on top of our <a href="#kernel-configs-about">default kernel configs</a>, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --config 'CONFIG_FORTIFY_SOURCE=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-modules --clean
./build-modules</pre>
</div>
</div>
<div class="paragraph">
<p>and then proceed as in <a href="#your-first-kernel-module-hack">Your first kernel module hack</a>.</p>
</div>
<div class="paragraph">
<p>You might often get way without rebuilding the kernel modules however.</p>
</div>
<div class="paragraph">
<p>To use an extra kernel config fragment file on top of our defaults, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
' &gt; data/myconfig
./build-linux --config-fragment 'data/myconfig'</pre>
</div>
</div>
<div class="paragraph">
<p>To use just your own exact <code>.config</code> instead of our defaults ones, use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --custom-config-file data/myconfig</pre>
</div>
</div>
<div class="paragraph">
<p>There is also a shortcut <code>--custom-config-file-gem5</code> to use the <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a>.</p>
</div>
<div class="paragraph">
<p>The following options can all be used together, sorted by decreasing config setting power precedence:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>--config</code></p>
</li>
<li>
<p><code>--config-fragment</code></p>
</li>
<li>
<p><code>--custom-config-file</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To do a clean menu config yourself and use that for the build, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --clean
./build-linux --custom-config-target menuconfig</pre>
</div>
</div>
<div class="paragraph">
<p>But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --no-configure</pre>
</div>
</div>
<div class="paragraph">
<p>So what you likely want to do instead is to save that as a new <code>defconfig</code> and use it later as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --no-configure --no-modules-install savedefconfig
cp "$(./getvar linux_build_dir)/defconfig" data/myconfig
./build-linux --custom-config-file data/myconfig</pre>
</div>
</div>
<div class="paragraph">
<p>You can also use other config generating targets such as <code>defconfig</code> with the same method as shown at: <a href="#linux-kernel-defconfig">Section 17.1.3.1.1, &#8220;Linux kernel defconfig&#8221;</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="find-the-kernel-config"><a class="anchor" href="#find-the-kernel-config"></a><a class="link" href="#find-the-kernel-config">17.1.2. Find the kernel config</a></h4>
<div class="paragraph">
<p>Get the build config in guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>zcat /proc/config.gz</pre>
</div>
</div>
<div class="paragraph">
<p>or with our shortcut:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./conf.sh</pre>
</div>
</div>
<div class="paragraph">
<p>or to conveniently grep for a specific option case insensitively:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./conf.sh ikconfig</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/conf.sh">rootfs_overlay/lkmc/conf.sh</a>.</p>
</div>
<div class="paragraph">
<p>This is enabled by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y</pre>
</div>
</div>
<div class="paragraph">
<p>From host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar linux_config)"</pre>
</div>
</div>
<div class="paragraph">
<p>Just for fun <a href="https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263" class="bare">https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./linux/scripts/extract-ikconfig "$(./getvar vmlinux)"</pre>
</div>
</div>
<div class="paragraph">
<p>although this can be useful when someone gives you a random image.</p>
</div>
</div>
<div class="sect3">
<h4 id="kernel-configs-about"><a class="anchor" href="#kernel-configs-about"></a><a class="link" href="#kernel-configs-about">17.1.3. About our Linux kernel configs</a></h4>
<div class="paragraph">
<p>By default, <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-linux">build-linux</a> generates a <code>.config</code> that is a mixture of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a base config extracted from Buildroot&#8217;s minimal per machine <code>.config</code>, which has the minimal options needed to boot as explained at: <a href="#buildroot-kernel-config">Section 17.1.3.1, &#8220;About Buildroot&#8217;s kernel configs&#8221;</a>.</p>
</li>
<li>
<p>small overlays put top of that</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To find out which kernel configs are being used exactly, simply run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --dry-run</pre>
</div>
</div>
<div class="paragraph">
<p>and look for the <code>merge_config.sh</code> call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: <a href="https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407" class="bare">https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407</a></p>
</div>
<div class="paragraph">
<p>For each arch, the base of our configs are named as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>linux_config/buildroot-&lt;arch&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>e.g.: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/buildroot-x86_64">linux_config/buildroot-x86_64</a>.</p>
</div>
<div class="paragraph">
<p>These configs are extracted directly from a Buildroot build with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/update-buildroot-kernel-configs">update-buildroot-kernel-configs</a>.</p>
</div>
<div class="paragraph">
<p>Note that Buildroot can <code>sed</code> override some of the configurations, e.g. it forces <code>CONFIG_BLK_DEV_INITRD=y</code> when <code>BR2_TARGET_ROOTFS_CPIO</code> is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with <code>make savedefconfig</code>: <a href="https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config" class="bare">https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config</a></p>
</div>
<div class="paragraph">
<p>On top of those, we add the following by default:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/min">linux_config/min</a>: see: <a href="#linux-kernel-min-config">Section 17.1.3.1.2, &#8220;Linux kernel min config&#8221;</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/default">linux_config/default</a>: other optional configs that we enable by default because they increase visibility, or expose some cool feature, and don&#8217;t significantly increase build time nor add significant runtime overhead</p>
<div class="paragraph">
<p>We have since observed that the kernel size itself is very bloated compared to <code>defconfig</code> as shown at: <a href="#linux-kernel-defconfig">Section 17.1.3.1.1, &#8220;Linux kernel defconfig&#8221;</a>.</p>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="buildroot-kernel-config"><a class="anchor" href="#buildroot-kernel-config"></a><a class="link" href="#buildroot-kernel-config">17.1.3.1. About Buildroot&#8217;s kernel configs</a></h5>
<div class="paragraph">
<p>To see Buildroot&#8217;s base configs, start from <a href="https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_x86_64_defconfig"><code>buildroot/configs/qemu_x86_64_defconfig</code></a>.</p>
</div>
<div class="paragraph">
<p>That file contains <code>BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-4.15.config"</code>, which points to the base config file used: <a href="https://github.com/buildroot/buildroot/blob/2018.05/board/qemu/x86_64/linux-4.15.config">board/qemu/x86_64/linux-4.15.config</a>.</p>
</div>
<div class="paragraph">
<p><code>arm</code>, on the other hand, uses <a href="https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_arm_vexpress_defconfig"><code>buildroot/configs/qemu_arm_vexpress_defconfig</code></a>, which contains <code>BR2_LINUX_KERNEL_DEFCONFIG="vexpress"</code>, and therefore just does a <code>make vexpress_defconfig</code>, and gets its config from the Linux kernel tree itself.</p>
</div>
<div class="sect5">
<h6 id="linux-kernel-defconfig"><a class="anchor" href="#linux-kernel-defconfig"></a><a class="link" href="#linux-kernel-defconfig">17.1.3.1.1. Linux kernel defconfig</a></h6>
<div class="paragraph">
<p>To boot <a href="https://stackoverflow.com/questions/41885015/what-exactly-does-linux-kernels-make-defconfig-do">defconfig</a> from disk on Linux and see a shell, all we need is these missing virtio options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
  --config CONFIG_VIRTIO_PCI=y \
  --config CONFIG_VIRTIO_BLK=y \
;
./run --linux-build-id defconfig</pre>
</div>
</div>
<div class="paragraph">
<p>Oh, and check this out:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>du -h \
  "$(./getvar vmlinux)" \
  "$(./getvar --linux-build-id defconfig vmlinux)" \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>360M    /path/to/linux-kernel-module-cheat/out/linux/default/x86_64/vmlinux
47M     /path/to/linux-kernel-module-cheat/out/linux/defconfig/x86_64/vmlinux</pre>
</div>
</div>
<div class="paragraph">
<p>Brutal. Where did we go wrong?</p>
</div>
<div class="paragraph">
<p>The extra virtio options are not needed if we use <a href="#initrd">initrd</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run --initrd --linux-build-id defconfig</pre>
</div>
</div>
<div class="paragraph">
<p>On aarch64, we can boot from initrd with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --arch aarch64 \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run \
  --arch aarch64 \
  --initrd \
  --linux-build-id defconfig \
  --memory 2G \
;</pre>
</div>
</div>
<div class="paragraph">
<p>We need the 2G of memory because the CPIO is 600MiB due to a humongous amount of loadable kernel modules!</p>
</div>
<div class="paragraph">
<p>In aarch64, the size situation is inverted from x86_64, and this can be seen on the vmlinux size as well:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>118M    /path/to/linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux
240M    /path/to/linux-kernel-module-cheat/out/linux/defconfig/aarch64/vmlinux</pre>
</div>
</div>
<div class="paragraph">
<p>So it seems that the ARM devs decided rather than creating a minimal config that boots QEMU, to try and make a single config that boots every board in existence. Terrible!</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://unix.stackexchange.com/questions/29439/compiling-the-kernel-with-default-configurations/204512#204512" class="bare">https://unix.stackexchange.com/questions/29439/compiling-the-kernel-with-default-configurations/204512#204512</a></p>
</div>
<div class="paragraph">
<p>Tested on 1e2b7f1e5e9e3073863dc17e25b2455c8ebdeadd + 1.</p>
</div>
</div>
<div class="sect5">
<h6 id="linux-kernel-min-config"><a class="anchor" href="#linux-kernel-min-config"></a><a class="link" href="#linux-kernel-min-config">17.1.3.1.2. Linux kernel min config</a></h6>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/min">linux_config/min</a> contains minimal tweaks required to boot gem5 or for using our slightly different QEMU command line options than Buildroot on all archs.</p>
</div>
<div class="paragraph">
<p>It is one of the default config fragments we use, as explained at: <a href="#kernel-configs-about">Section 17.1.3, &#8220;About our Linux kernel configs&#8221;</a>&gt;.</p>
</div>
<div class="paragraph">
<p>Having the same config working for both QEMU and gem5 (oh, the hours of bisection) means that you can deal with functional matters in QEMU, which runs much faster, and switch to gem5 only for performance issues.</p>
</div>
<div class="paragraph">
<p>We can build just with <code>min</code> on top of the base config with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/min \
  --custom-config-file linux_config/buildroot-aarch64 \
  --linux-build-id min \
;</pre>
</div>
</div>
<div class="paragraph">
<p>vmlinux had a very similar size to the default. It seems that <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/buildroot-aarch64">linux_config/buildroot-aarch64</a> contains or implies most <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/default">linux_config/default</a> options already? TODO: that seems odd, really?</p>
</div>
<div class="paragraph">
<p>Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="notable-alternate-gem5-kernel-configs"><a class="anchor" href="#notable-alternate-gem5-kernel-configs"></a><a class="link" href="#notable-alternate-gem5-kernel-configs">17.1.3.2. Notable alternate gem5 kernel configs</a></h5>
<div class="paragraph">
<p>Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>arm</code> and <code>aarch64</code> configs present in the official ARM gem5 Linux kernel fork as described at: <a href="#gem5-arm-linux-kernel-patches">Section 24.9, &#8220;gem5 arm Linux kernel patches&#8221;</a>. Some of the configs present there are added by the patches.</p>
</li>
<li>
<p>Jason&#8217;s magic <code>x86_64</code> config: <a href="http://web.archive.org/web/20171229121642/http://www.lowepower.com/jason/files/config" class="bare">http://web.archive.org/web/20171229121642/http://www.lowepower.com/jason/files/config</a> which is referenced at: <a href="http://web.archive.org/web/20171229121525/http://www.lowepower.com/jason/setting-up-gem5-full-system.html" class="bare">http://web.archive.org/web/20171229121525/http://www.lowepower.com/jason/setting-up-gem5-full-system.html</a>. QEMU boots with that by removing <code># CONFIG_VIRTIO_PCI is not set</code>.</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel-version"><a class="anchor" href="#kernel-version"></a><a class="link" href="#kernel-version">17.2. Kernel version</a></h3>
<div class="sect3">
<h4 id="find-the-kernel-version"><a class="anchor" href="#find-the-kernel-version"></a><a class="link" href="#find-the-kernel-version">17.2.1. Find the kernel version</a></h4>
<div class="paragraph">
<p>We try to use the latest possible kernel major release version.</p>
</div>
<div class="paragraph">
<p>In QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/version</pre>
</div>
</div>
<div class="paragraph">
<p>or in the source:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd "$(./getvar linux_source_dir)"
git log | grep -E '    Linux [0-9]+\.' | head</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="update-the-linux-kernel"><a class="anchor" href="#update-the-linux-kernel"></a><a class="link" href="#update-the-linux-kernel">17.2.2. Update the Linux kernel</a></h4>
<div class="paragraph">
<p>During update all you kernel modules may break since the kernel API is not stable.</p>
</div>
<div class="paragraph">
<p>They are usually trivial breaks of things moving around headers or to sub-structs.</p>
</div>
<div class="paragraph">
<p>The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.</p>
</div>
<div class="paragraph">
<p>This backwards compatibility is just awesome, it makes getting and running the latest master painless.</p>
</div>
<div class="paragraph">
<p>This also makes this repo the perfect setup to develop the Linux kernel.</p>
</div>
<div class="paragraph">
<p>In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: <a href="#bisection">Section 38.17, &#8220;Bisection&#8221;</a>.</p>
</div>
<div class="sect4">
<h5 id="update-the-linux-kernel-lkmc-procedure"><a class="anchor" href="#update-the-linux-kernel-lkmc-procedure"></a><a class="link" href="#update-the-linux-kernel-lkmc-procedure">17.2.2.1. Update the Linux kernel LKMC procedure</a></h5>
<div class="paragraph">
<p>First, use use the branching procedure described at: <a href="#update-a-forked-submodule">Section 38.18, &#8220;Update a forked submodule&#8221;</a></p>
</div>
<div class="paragraph">
<p>Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: <a href="#test-this-repo">Section 38.16, &#8220;Test this repo&#8221;</a>. The only tests that can be skipped are essentially the <a href="#baremetal">Baremetal</a> tests.</p>
</div>
<div class="paragraph">
<p>Before comitting, don&#8217;t forget to update:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the <code>linux_kernel_version</code> constant in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/common.py">common.py</a></p>
</li>
<li>
<p>the tagline of this repository on:</p>
<div class="ulist">
<ul>
<li>
<p>this README</p>
</li>
<li>
<p>the GitHub project description</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="downgrade-the-linux-kernel"><a class="anchor" href="#downgrade-the-linux-kernel"></a><a class="link" href="#downgrade-the-linux-kernel">17.2.3. Downgrade the Linux kernel</a></h4>
<div class="paragraph">
<p>The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.</p>
</div>
<div class="paragraph">
<p>The default Linux kernel version is bumped in Buildroot with commit messages of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>linux: bump default to version 4.9.6</pre>
</div>
</div>
<div class="paragraph">
<p>So you can try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git log --grep 'linux: bump default to version'</pre>
</div>
</div>
<div class="paragraph">
<p>Those commits change <code>BR2_LINUX_KERNEL_LATEST_VERSION</code> in <code>/linux/Config.in</code>.</p>
</div>
<div class="paragraph">
<p>You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.</p>
</div>
<div class="paragraph">
<p>Finally, after downgrading Buildroot, if something does not work, you might also have to make some changes to how this repo uses Buildroot, as the Buildroot configuration options might have changed.</p>
</div>
<div class="paragraph">
<p>We don&#8217;t expect those changes to be very difficult. A good way to approach the task is to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>do a dry run build to get the equivalent Bash commands used:</p>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --dry-run</pre>
</div>
</div>
</li>
<li>
<p>build the Buildroot documentation for the version you are going to use, and check if all Buildroot build commands make sense there</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, if you spot an option that is wrong, some grepping in this repo should quickly point you to the code you need to modify.</p>
</div>
<div class="paragraph">
<p>It also possible that you will need to apply some patches from newer Buildroot versions for it to build, due to incompatibilities with the host Ubuntu packages and that Buildroot version. Just read the error message, and try:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>git log master&#8201;&#8212;&#8201;packages/&lt;pkg&gt;</code></p>
</li>
<li>
<p>Google the error message for mailing list hits</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Successful port reports:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>v3.18: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/39#issuecomment-438525481" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/39#issuecomment-438525481</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel-command-line-parameters"><a class="anchor" href="#kernel-command-line-parameters"></a><a class="link" href="#kernel-command-line-parameters">17.3. Kernel command line parameters</a></h3>
<div class="paragraph">
<p>Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the <code>execve</code> system call does to userland processes.</p>
</div>
<div class="paragraph">
<p>This allows us to control the behaviour of the kernel without rebuilding anything.</p>
</div>
<div class="paragraph">
<p>With QEMU, QEMU itself acts as the bootloader, and provides the <code>-append</code> option and we expose it through <code>./run --kernel-cli</code>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'foo bar'</pre>
</div>
</div>
<div class="paragraph">
<p>Then inside the host, you can check which options were given with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/cmdline</pre>
</div>
</div>
<div class="paragraph">
<p>They are also printed at the beginning of the boot message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dmesg | grep "Command line"</pre>
</div>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/48601/how-to-display-the-linux-kernel-command-line-parameters-given-for-the-current-bo" class="bare">https://unix.stackexchange.com/questions/48601/how-to-display-the-linux-kernel-command-line-parameters-given-for-the-current-bo</a></p>
</li>
<li>
<p><a href="https://askubuntu.com/questions/32654/how-do-i-find-the-boot-parameters-used-by-the-running-kernel" class="bare">https://askubuntu.com/questions/32654/how-do-i-find-the-boot-parameters-used-by-the-running-kernel</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The arguments are documented in the kernel documentation: <a href="https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html" class="bare">https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html</a></p>
</div>
<div class="paragraph">
<p>When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>GRUB configuration files: <a href="https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter" class="bare">https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter</a></p>
</li>
<li>
<p>Raspberry pi <code>/boot/cmdline.txt</code> on a magic partition: <a href="https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly" class="bare">https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="kernel-command-line-parameters-escaping"><a class="anchor" href="#kernel-command-line-parameters-escaping"></a><a class="link" href="#kernel-command-line-parameters-escaping">17.3.1. Kernel command line parameters escaping</a></h4>
<div class="paragraph">
<p>Double quotes can be used to escape spaces as in <code>opt="a b"</code>, but double quotes themselves cannot be escaped, e.g. <code>opt"a\"b"</code></p>
</div>
<div class="paragraph">
<p>This even lead us to use base64 encoding with <code>--eval</code>!</p>
</div>
</div>
<div class="sect3">
<h4 id="kernel-command-line-parameters-definition-points"><a class="anchor" href="#kernel-command-line-parameters-definition-points"></a><a class="link" href="#kernel-command-line-parameters-definition-points">17.3.2. Kernel command line parameters definition points</a></h4>
<div class="paragraph">
<p>There are two methods:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>__setup</code> as in:</p>
<div class="literalblock">
<div class="content">
<pre>__setup("console=", console_setup);</pre>
</div>
</div>
</li>
<li>
<p><code>core_param</code> as in:</p>
<div class="literalblock">
<div class="content">
<pre>core_param(panic, panic_timeout, int, 0644);</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>core_param</code> suggests how they are different:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/**
 * core_param - define a historical core kernel parameter.

...

 * core_param is just like module_param(), but cannot be modular and
 * doesn't add a prefix (such as "printk.").  This is for compatibility
 * with __setup(), and it makes sense as truly core parameters aren't
 * tied to the particular file they're in.
 */</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="rw"><a class="anchor" href="#rw"></a><a class="link" href="#rw">17.3.3. rw</a></h4>
<div class="paragraph">
<p>By default, the Linux kernel mounts the root filesystem as readonly. TODO rationale?</p>
</div>
<div class="paragraph">
<p>This cannot be observed in the default BusyBox init, because by default our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/inittab">rootfs_overlay/etc/inittab</a> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/bin/mount -o remount,rw /</pre>
</div>
</div>
<div class="paragraph">
<p>Analogously, Ubuntu 18.04 does in its fstab something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>UUID=/dev/sda1 / ext4 errors=remount-ro 0 1</pre>
</div>
</div>
<div class="paragraph">
<p>which uses default mount <code>rw</code> flags.</p>
</div>
<div class="paragraph">
<p>We have however removed those setups init setups to keep things more minimal, and replaced them with the <code>rw</code> kernel boot parameter makes the root mounted as writable.</p>
</div>
<div class="paragraph">
<p>To observe the default readonly behaviour, hack the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> script to remove <a href="#replace-init">replace init</a>, and then run on a raw shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/bin/sh'</pre>
</div>
</div>
<div class="paragraph">
<p>Now try to do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>touch a</pre>
</div>
</div>
<div class="paragraph">
<p>which fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>touch: a: Read-only file system</pre>
</div>
</div>
<div class="paragraph">
<p>We can also observe the read-onlyness with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mount -t proc /proc
mount</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dev/root on / type ext2 (ro,relatime,block_validity,barrier,user_xattr)</pre>
</div>
</div>
<div class="paragraph">
<p>and so it is Read Only as shown by <code>ro</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="norandmaps"><a class="anchor" href="#norandmaps"></a><a class="link" href="#norandmaps">17.3.4. norandmaps</a></h4>
<div class="paragraph">
<p>Disable userland address space randomization. Test it out by running <a href="#rand-check-out">rand_check.out</a> twice:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './linux/rand_check.out;./linux/poweroff.out'
./run --eval-after './linux/rand_check.out;./linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>If we remove it from our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> script by hacking it up, the addresses shown by <code>linux/rand_check.out</code> vary across boots.</p>
</div>
<div class="paragraph">
<p>Equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 0 &gt; /proc/sys/kernel/randomize_va_space</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="printk"><a class="anchor" href="#printk"></a><a class="link" href="#printk">17.4. printk</a></h3>
<div class="paragraph">
<p><code>printk</code> is the most simple and widely used way of getting information from the kernel, so you should familiarize yourself with its basic configuration.</p>
</div>
<div class="paragraph">
<p>We use <code>printk</code> a lot in our kernel modules, and it shows on the terminal by default, along with stdout and what you type.</p>
</div>
<div class="paragraph">
<p>Hide all <code>printk</code> messages:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dmesg -n 1</pre>
</div>
</div>
<div class="paragraph">
<p>or equivalently:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/kernel/printk</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console" class="bare">https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console</a></p>
</div>
<div class="paragraph">
<p>Do it with a <a href="#kernel-command-line-parameters">Kernel command line parameters</a> to affect the boot itself:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'loglevel=5'</pre>
</div>
</div>
<div class="paragraph">
<p>and now only boot warning messages or worse show, which is useful to identify problems.</p>
</div>
<div class="paragraph">
<p>Our default <code>printk</code> format is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;LEVEL&gt;[TIMESTAMP] MESSAGE</pre>
</div>
</div>
<div class="paragraph">
<p>e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    2.979121] Freeing unused kernel memory: 2024K</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>LEVEL</code>: higher means less serious</p>
</li>
<li>
<p><code>TIMESTAMP</code>: seconds since boot</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This format is selected by the following boot options:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>console_msg_format=syslog</code>: add the <code>&lt;LEVEL&gt;</code> part. Added in v4.16.</p>
</li>
<li>
<p><code>printk.time=y</code>: add the <code>[TIMESTAMP]</code> part</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The debug highest level is a bit more magic, see: <a href="#pr-debug">Section 17.4.3, &#8220;pr_debug&#8221;</a> for more info.</p>
</div>
<div class="sect3">
<h4 id="procsyskernelprintk"><a class="anchor" href="#procsyskernelprintk"></a><a class="link" href="#procsyskernelprintk">17.4.1. /proc/sys/kernel/printk</a></h4>
<div class="paragraph">
<p>The current printk level can be obtained with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/sys/kernel/printk</pre>
</div>
</div>
<div class="paragraph">
<p>As of <code>87e846fc1f9c57840e143513ebd69c638bd37aa8</code> this prints:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>7       4       1       7</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>7</code>: current log level, modifiable by previously mentioned methods</p>
</li>
<li>
<p><code>4</code>: documented as: "printk&#8217;s without a loglevel use this": TODO what does that mean, how to call <code>printk</code> without a log level?</p>
</li>
<li>
<p><code>1</code>: minimum log level that still prints something (<code>0</code> prints nothing)</p>
</li>
<li>
<p><code>7</code>: default log level</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We start at the boot time default after boot by default, as can be seen from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod myprintk.ko</pre>
</div>
</div>
<div class="paragraph">
<p>which outputs something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;1&gt;[   12.494429] pr_alert
&lt;2&gt;[   12.494666] pr_crit
&lt;3&gt;[   12.494823] pr_err
&lt;4&gt;[   12.494911] pr_warning
&lt;5&gt;[   12.495170] pr_notice
&lt;6&gt;[   12.495327] pr_info</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/myprintk.c">kernel_modules/myprintk.c</a></p>
</div>
<div class="paragraph">
<p>This proc entry is defined at: <a href="https://github.com/torvalds/linux/blob/v5.1/kernel/sysctl.c#L839" class="bare">https://github.com/torvalds/linux/blob/v5.1/kernel/sysctl.c#L839</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>#if defined CONFIG_PRINTK
	{
		.procname	= "printk",
		.data		= &amp;console_loglevel,
		.maxlen		= 4*sizeof(int),
		.mode		= 0644,
		.proc_handler	= proc_dointvec,
	},</pre>
</div>
</div>
<div class="paragraph">
<p>which teaches us that printk can be completely disabled at compile time:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>config PRINTK
	default y
	bool "Enable support for printk" if EXPERT
	select IRQ_WORK
	help
	  This option enables normal printk support. Removing it
	  eliminates most of the message strings from the kernel image
	  and makes the kernel more or less silent. As this makes it
	  very difficult to diagnose system problems, saying N here is
	  strongly discouraged.</pre>
</div>
</div>
<div class="paragraph">
<p><code>console_loglevel</code> is defined at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#define console_loglevel (console_printk[0])</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>console_printk</code> is an array with 4 ints:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>int console_printk[4] = {
	CONSOLE_LOGLEVEL_DEFAULT,	/* console_loglevel */
	MESSAGE_LOGLEVEL_DEFAULT,	/* default_message_loglevel */
	CONSOLE_LOGLEVEL_MIN,		/* minimum_console_loglevel */
	CONSOLE_LOGLEVEL_DEFAULT,	/* default_console_loglevel */
};</pre>
</div>
</div>
<div class="paragraph">
<p>and then we see that the default is configurable with <code>CONFIG_CONSOLE_LOGLEVEL_DEFAULT</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/*
 * Default used to be hard-coded at 7, quiet used to be hardcoded at 4,
 * we're now allowing both to be set from kernel config.
 */
#define CONSOLE_LOGLEVEL_DEFAULT CONFIG_CONSOLE_LOGLEVEL_DEFAULT
#define CONSOLE_LOGLEVEL_QUIET	 CONFIG_CONSOLE_LOGLEVEL_QUIET</pre>
</div>
</div>
<div class="paragraph">
<p>The message loglevel default is explained at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* printk's without a loglevel use this.. */
#define MESSAGE_LOGLEVEL_DEFAULT CONFIG_MESSAGE_LOGLEVEL_DEFAULT</pre>
</div>
</div>
<div class="paragraph">
<p>The min is just hardcoded to one as you would expect, with some amazing kernel comedy around it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* We show everything that is MORE important than this.. */
#define CONSOLE_LOGLEVEL_SILENT  0 /* Mum's the word */
#define CONSOLE_LOGLEVEL_MIN	 1 /* Minimum loglevel we let people use */
#define CONSOLE_LOGLEVEL_DEBUG	10 /* issue debug messages */
#define CONSOLE_LOGLEVEL_MOTORMOUTH 15	/* You can't shut this one up */</pre>
</div>
</div>
<div class="paragraph">
<p>We then also learn about the useless <code>quiet</code> and <code>debug</code> kernel parameters at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>config CONSOLE_LOGLEVEL_QUIET
	int "quiet console loglevel (1-15)"
	range 1 15
	default "4"
	help
	  loglevel to use when "quiet" is passed on the kernel commandline.

	  When "quiet" is passed on the kernel commandline this loglevel
	  will be used as the loglevel. IOW passing "quiet" will be the
	  equivalent of passing "loglevel=&lt;CONSOLE_LOGLEVEL_QUIET&gt;"</pre>
</div>
</div>
<div class="paragraph">
<p>which explains the useless reason why that number is special. This is implemented at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static int __init debug_kernel(char *str)
{
	console_loglevel = CONSOLE_LOGLEVEL_DEBUG;
	return 0;
}

static int __init quiet_kernel(char *str)
{
	console_loglevel = CONSOLE_LOGLEVEL_QUIET;
	return 0;
}

early_param("debug", debug_kernel);
early_param("quiet", quiet_kernel);</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="ignore-loglevel"><a class="anchor" href="#ignore-loglevel"></a><a class="link" href="#ignore-loglevel">17.4.2. ignore_loglevel</a></h4>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'ignore_loglevel'</pre>
</div>
</div>
<div class="paragraph">
<p>enables all log levels, and is basically the same as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'loglevel=8'</pre>
</div>
</div>
<div class="paragraph">
<p>except that you don&#8217;t need to know what is the maximum level.</p>
</div>
</div>
<div class="sect3">
<h4 id="pr-debug"><a class="anchor" href="#pr-debug"></a><a class="link" href="#pr-debug">17.4.3. pr_debug</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405" class="bare">https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405</a></p>
</div>
<div class="paragraph">
<p>Debug messages are not printable by default without recompiling.</p>
</div>
<div class="paragraph">
<p>But the awesome <code>CONFIG_DYNAMIC_DEBUG=y</code> option which we enable by default allows us to do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 8 &gt; /proc/sys/kernel/printk
echo 'file kernel/module.c +p' &gt; /sys/kernel/debug/dynamic_debug/control
./linux/myinsmod.out hello.ko</pre>
</div>
</div>
<div class="paragraph">
<p>and we have a shortcut at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./pr_debug.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/pr_debug.sh">rootfs_overlay/lkmc/pr_debug.sh</a>.</p>
</div>
<div class="paragraph">
<p>Syntax: <a href="https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html" class="bare">https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html</a></p>
</div>
<div class="paragraph">
<p>Wildcards are also accepted, e.g. enable all messages from all files:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 'file * +p' &gt; /sys/kernel/debug/dynamic_debug/control</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: why is this not working:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 'func sys_init_module +p' &gt; /sys/kernel/debug/dynamic_debug/control</pre>
</div>
</div>
<div class="paragraph">
<p>Enable messages in specific modules:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 8 &gt; /proc/sys/kernel/printk
echo 'module myprintk +p' &gt; /sys/kernel/debug/dynamic_debug/control
insmod myprintk.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/myprintk.c">kernel_modules/myprintk.c</a></p>
</div>
<div class="paragraph">
<p>This outputs the <code>pr_debug</code> message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printk debug</pre>
</div>
</div>
<div class="paragraph">
<p>but TODO: it also shows debug messages even without enabling them explicitly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 8 &gt; /proc/sys/kernel/printk
insmod myprintk.ko</pre>
</div>
</div>
<div class="paragraph">
<p>and it shows as enabled:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># grep myprintk /sys/kernel/debug/dynamic_debug/control
/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c:12 [myprintk]myinit =p "pr_debug\012"</pre>
</div>
</div>
<div class="paragraph">
<p>Enable <code>pr_debug</code> for boot messages as well, before we can reach userland and write to <code>/proc</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'dyndbg="file * +p" loglevel=8'</pre>
</div>
</div>
<div class="paragraph">
<p>Get ready for the noisiest boot ever, I think it overflows the <code>printk</code> buffer and funny things happen.</p>
</div>
<div class="sect4">
<h5 id="pr-debug-is-different-from-printk-kern-debug"><a class="anchor" href="#pr-debug-is-different-from-printk-kern-debug"></a><a class="link" href="#pr-debug-is-different-from-printk-kern-debug">17.4.3.1. pr_debug != printk(KERN_DEBUG</a></h5>
<div class="paragraph">
<p>When <code>CONFIG_DYNAMIC_DEBUG</code> is set,  <code>printk(KERN_DEBUG</code> is not the exact same as <code>pr_debug(</code> since <code>printk(KERN_DEBUG</code> messages are visible with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'initcall_debug logleve=8'</pre>
</div>
</div>
<div class="paragraph">
<p>which outputs lines of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;7&gt;[    1.756680] calling  clk_disable_unused+0x0/0x130 @ 1
&lt;7&gt;[    1.757003] initcall clk_disable_unused+0x0/0x130 returned 0 after 111 usecs</pre>
</div>
</div>
<div class="paragraph">
<p>which are <code>printk(KERN_DEBUG</code> inside <code>init/main.c</code> in v4.16.</p>
</div>
<div class="paragraph">
<p>Mentioned at: <a href="https://stackoverflow.com/questions/37272109/how-to-get-details-of-all-modules-drivers-got-initialized-probed-during-kernel-b" class="bare">https://stackoverflow.com/questions/37272109/how-to-get-details-of-all-modules-drivers-got-initialized-probed-during-kernel-b</a></p>
</div>
<div class="paragraph">
<p>This likely comes from the ifdef split at <code>init/main.c</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* If you are writing a driver, please use dev_dbg instead */
#if defined(CONFIG_DYNAMIC_DEBUG)
#include &lt;linux/dynamic_debug.h&gt;

/* dynamic_pr_debug() uses pr_fmt() internally so we don't need it here */
#define pr_debug(fmt, ...) \
    dynamic_pr_debug(fmt, ##__VA_ARGS__)
#elif defined(DEBUG)
#define pr_debug(fmt, ...) \
    printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#else
#define pr_debug(fmt, ...) \
    no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#endif</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel-module-apis"><a class="anchor" href="#kernel-module-apis"></a><a class="link" href="#kernel-module-apis">17.5. Kernel module APIs</a></h3>
<div class="sect3">
<h4 id="kernel-module-parameters"><a class="anchor" href="#kernel-module-parameters"></a><a class="link" href="#kernel-module-parameters">17.5.1. Kernel module parameters</a></h4>
<div class="paragraph">
<p>The Linux kernel allows passing module parameters at insertion time <a href="#myinsmod">through the <code>init_module</code> and <code>finit_module</code> system calls</a>.</p>
</div>
<div class="paragraph">
<p>The <code>insmod</code> tool exposes that as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod params.ko i=3 j=4</pre>
</div>
</div>
<div class="paragraph">
<p>Parameters are declared in the module as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static u32 i = 0;
module_param(i, int, S_IRUSR | S_IWUSR);
MODULE_PARM_DESC(i, "my favorite int");</pre>
</div>
</div>
<div class="paragraph">
<p>Automated test:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./params.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/params.c">kernel_modules/params.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/params.sh">rootfs_overlay/lkmc/params.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>As shown in the example, module parameters can also be read and modified at runtime from <a href="#sysfs">sysfs</a>.</p>
</div>
<div class="paragraph">
<p>We can obtain the help text of the parameters with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modinfo params.ko</pre>
</div>
</div>
<div class="paragraph">
<p>The output contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>parm:           j:my second favorite int
parm:           i:my favorite int</pre>
</div>
</div>
<div class="sect4">
<h5 id="modprobe-conf"><a class="anchor" href="#modprobe-conf"></a><a class="link" href="#modprobe-conf">17.5.1.1. modprobe.conf</a></h5>
<div class="paragraph">
<p><a href="#modprobe">modprobe</a> insertion can also set default parameters via the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/modprobe.conf"><code>/etc/modprobe.conf</code></a> file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modprobe params
cat /sys/kernel/debug/lkmc_params</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>12 34</pre>
</div>
</div>
<div class="paragraph">
<p>This is specially important when loading modules with <a href="#kernel-module-dependencies">Kernel module dependencies</a> or else we would have no opportunity of passing those.</p>
</div>
<div class="paragraph">
<p><code>modprobe.conf</code> doesn&#8217;t actually insmod anything for us: <a href="https://superuser.com/questions/397842/automatically-load-kernel-module-at-boot-angstrom/1267464#1267464" class="bare">https://superuser.com/questions/397842/automatically-load-kernel-module-at-boot-angstrom/1267464#1267464</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="kernel-module-dependencies"><a class="anchor" href="#kernel-module-dependencies"></a><a class="link" href="#kernel-module-dependencies">17.5.2. Kernel module dependencies</a></h4>
<div class="paragraph">
<p>One module can depend on symbols of another module that are exported with <code>EXPORT_SYMBOL</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./dep.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/dep.c">kernel_modules/dep.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/dep2.c">kernel_modules/dep2.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/dep.sh">rootfs_overlay/lkmc/dep.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The kernel deduces dependencies based on the <code>EXPORT_SYMBOL</code> that each module uses.</p>
</div>
<div class="paragraph">
<p>Symbols exported by <code>EXPORT_SYMBOL</code> can be seen with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod dep.ko
grep lkmc_dep /proc/kallsyms</pre>
</div>
</div>
<div class="paragraph">
<p>sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ffffffffc0001030 r __ksymtab_lkmc_dep   [dep]
ffffffffc000104d r __kstrtab_lkmc_dep   [dep]
ffffffffc0002300 B lkmc_dep     [dep]</pre>
</div>
</div>
<div class="paragraph">
<p>This requires <code>CONFIG_KALLSYMS_ALL=y</code>.</p>
</div>
<div class="paragraph">
<p>Dependency information is stored by the kernel module build system in the <code>.ko</code> files' <a href="#module-info">MODULE_INFO</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modinfo dep2.ko</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>depends:        dep</pre>
</div>
</div>
<div class="paragraph">
<p>We can double check with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>strings 3 dep2.ko | grep -E 'depends'</pre>
</div>
</div>
<div class="paragraph">
<p>The output contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>depends=dep</pre>
</div>
</div>
<div class="paragraph">
<p>Module dependencies are also stored at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd /lib/module/*
grep dep modules.dep</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>extra/dep2.ko: extra/dep.ko
extra/dep.ko:</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what for, and at which point point does Buildroot / BusyBox generate that file?</p>
</div>
<div class="sect4">
<h5 id="kernel-module-dependencies-with-modprobe"><a class="anchor" href="#kernel-module-dependencies-with-modprobe"></a><a class="link" href="#kernel-module-dependencies-with-modprobe">17.5.2.1. Kernel module dependencies with modprobe</a></h5>
<div class="paragraph">
<p>Unlike <code>insmod</code>, <a href="#modprobe">modprobe</a> deals with kernel module dependencies for us.</p>
</div>
<div class="paragraph">
<p>First get <a href="#kernel-modules-buildroot-package">kernel_modules buildroot package</a> working.</p>
</div>
<div class="paragraph">
<p>Then, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modprobe buildroot_dep2</pre>
</div>
</div>
<div class="paragraph">
<p>outputs to dmesg:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>42</pre>
</div>
</div>
<div class="paragraph">
<p>and then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lsmod</pre>
</div>
</div>
<div class="paragraph">
<p>outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Module                  Size  Used by    Tainted: G
buildroot_dep2         16384  0
buildroot_dep          16384  1 buildroot_dep2</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/kernel_modules/buildroot_dep.c">buildroot_packages/kernel_modules/buildroot_dep.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/kernel_modules/buildroot_dep2.c">buildroot_packages/kernel_modules/buildroot_dep2.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Removal also removes required modules that have zero usage count:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modprobe -r buildroot_dep2</pre>
</div>
</div>
<div class="paragraph">
<p><code>modprobe</code> uses information from the <code>modules.dep</code> file to decide the required dependencies. That file contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>extra/buildroot_dep2.ko: extra/buildroot_dep.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/20070/whats-the-difference-between-insmod-and-modprobe" class="bare">https://askubuntu.com/questions/20070/whats-the-difference-between-insmod-and-modprobe</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/22891705/whats-the-difference-between-insmod-and-modprobe" class="bare">https://stackoverflow.com/questions/22891705/whats-the-difference-between-insmod-and-modprobe</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="module-info"><a class="anchor" href="#module-info"></a><a class="link" href="#module-info">17.5.3. MODULE_INFO</a></h4>
<div class="paragraph">
<p>Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the <code>THIS_MODULE</code> <code>struct module</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod module_info.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Dmesg output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>name = module_info
version = 1.0</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/module_info.c">kernel_modules/module_info.c</a></p>
</div>
<div class="paragraph">
<p>Some of those are also present on sysfs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /sys/module/module_info/version</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>1.0</pre>
</div>
</div>
<div class="paragraph">
<p>And we can also observe them with the <code>modinfo</code> command line utility:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modinfo module_info.ko</pre>
</div>
</div>
<div class="paragraph">
<p>sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>filename:       module_info.ko
license:        GPL
version:        1.0
srcversion:     AF3DE8A8CFCDEB6B00E35B6
depends:
vermagic:       4.17.0 SMP mod_unload modversions</pre>
</div>
</div>
<div class="paragraph">
<p>Module information is stored in a special <code>.modinfo</code> section of the ELF file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- -SW "$(./getvar kernel_modules_build_subdir)/module_info.ko"</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  [ 5] .modinfo          PROGBITS        0000000000000000 0000d8 000096 00   A  0   0  8</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- -x .modinfo "$(./getvar kernel_modules_build_subdir)/module_info.ko"</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  0x00000000 6c696365 6e73653d 47504c00 76657273 license=GPL.vers
  0x00000010 696f6e3d 312e3000 61736466 3d717765 ion=1.0.asdf=qwe
  0x00000020 72000000 00000000 73726376 65727369 r.......srcversi
  0x00000030 6f6e3d41 46334445 38413843 46434445 on=AF3DE8A8CFCDE
  0x00000040 42364230 30453335 42360000 00000000 B6B00E35B6......
  0x00000050 64657065 6e64733d 006e616d 653d6d6f depends=.name=mo
  0x00000060 64756c65 5f696e66 6f007665 726d6167 dule_info.vermag
  0x00000070 69633d34 2e31372e 3020534d 50206d6f ic=4.17.0 SMP mo
  0x00000080 645f756e 6c6f6164 206d6f64 76657273 d_unload modvers
  0x00000090 696f6e73 2000                       ions .</pre>
</div>
</div>
<div class="paragraph">
<p>I think a dedicated section is used to allow the Linux kernel and command line tools to easily parse that information from the ELF file as we&#8217;ve done with <code>readelf</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/19467150/significance-of-this-module-in-linux-driver/49812248#49812248" class="bare">https://stackoverflow.com/questions/19467150/significance-of-this-module-in-linux-driver/49812248#49812248</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/4839024/how-to-find-the-version-of-a-compiled-kernel-module/42556565#42556565" class="bare">https://stackoverflow.com/questions/4839024/how-to-find-the-version-of-a-compiled-kernel-module/42556565#42556565</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/238167/how-to-understand-the-modinfo-output" class="bare">https://unix.stackexchange.com/questions/238167/how-to-understand-the-modinfo-output</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="vermagic"><a class="anchor" href="#vermagic"></a><a class="link" href="#vermagic">17.5.4. vermagic</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/vermagic.c">kernel_modules/vermagic.c</a></p>
</div>
<div class="paragraph">
<p>As of kernel v5.8, you can&#8217;t use <code>VERMAGIC_STRING</code> string from modules anymore as per: <a href="https://github.com/cirosantilli/linux/commit/51161bfc66a68d21f13d15a689b3ea7980457790" class="bare">https://github.com/cirosantilli/linux/commit/51161bfc66a68d21f13d15a689b3ea7980457790</a>. So instead we just showcase <code>init_utsname</code>.</p>
</div>
<div class="paragraph">
<p>Sample insmod output as of LKMC fa8c2ee521ea83a74a2300e7a3be9f9ab86e2cb6 + 1 aarch64:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[   25.180697] sysname    = Linux
&lt;6&gt;[   25.180697] nodename   = buildroot
&lt;6&gt;[   25.180697] release    = 5.9.2
&lt;6&gt;[   25.180697] version    = #1 SMP Thu Jan 1 00:00:00 UTC 1970
&lt;6&gt;[   25.180697] machine    = aarch64
&lt;6&gt;[   25.180697] domainname = (none)</pre>
</div>
</div>
<div class="paragraph">
<p>Vermagic is a magic string present in the kernel and previously visible in <a href="#module-info">MODULE_INFO</a> on kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod vermagic.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Possible dmesg output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>VERMAGIC_STRING = 4.17.0 SMP mod_unload modversions</pre>
</div>
</div>
<div class="paragraph">
<p>If we artificially create a mismatch with <code>MODULE_INFO(vermagic</code>, the insmod fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod: can't insert 'vermagic_fail.ko': invalid module format</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>dmesg</code> says the expected and found vermagic found:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vermagic_fail: version magic 'asdfqwer' should be '4.17.0 SMP mod_unload modversions '</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/vermagic_fail.c">kernel_modules/vermagic_fail.c</a></p>
</div>
<div class="paragraph">
<p>The kernel&#8217;s vermagic is defined based on compile time configurations at <a href="https://github.com/torvalds/linux/blob/v4.17/include/linux/vermagic.h#L35">include/linux/vermagic.h</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#define VERMAGIC_STRING                                                 \
        UTS_RELEASE " "                                                 \
        MODULE_VERMAGIC_SMP MODULE_VERMAGIC_PREEMPT                     \
        MODULE_VERMAGIC_MODULE_UNLOAD MODULE_VERMAGIC_MODVERSIONS       \
        MODULE_ARCH_VERMAGIC                                            \
        MODULE_RANDSTRUCT_PLUGIN</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>SMP</code> part of the string for example is defined on the same file based on the value of <code>CONFIG_SMP</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#ifdef CONFIG_SMP
#define MODULE_VERMAGIC_SMP "SMP "
#else
#define MODULE_VERMAGIC_SMP ""</pre>
</div>
</div>
<div class="paragraph">
<p>TODO how to get the vermagic from running kernel from userland? <a href="https://lists.kernelnewbies.org/pipermail/kernelnewbies/2012-October/006306.html" class="bare">https://lists.kernelnewbies.org/pipermail/kernelnewbies/2012-October/006306.html</a></p>
</div>
<div class="paragraph">
<p><a href="#kmod-modprobe">kmod modprobe</a> has a flag to skip the vermagic check:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--force-modversion</pre>
</div>
</div>
<div class="paragraph">
<p>This option just strips <code>modversion</code> information from the module before loading, so it is not a kernel feature.</p>
</div>
</div>
<div class="sect3">
<h4 id="init-module"><a class="anchor" href="#init-module"></a><a class="link" href="#init-module">17.5.5. init_module</a></h4>
<div class="paragraph">
<p><code>init_module</code> and <code>cleanup_module</code> are an older alternative to the <code>module_init</code> and <code>module_exit</code> macros:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod init_module.ko
rmmod init_module</pre>
</div>
</div>
<div class="paragraph">
<p>Dmesg output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>init_module
cleanup_module</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/init_module.c">kernel_modules/init_module.c</a></p>
</div>
<div class="paragraph">
<p>TODO why were <code>module_init</code> and <code>module_exit</code> created? <a href="https://stackoverflow.com/questions/3218320/what-is-the-difference-between-module-init-and-init-module-in-a-linux-kernel-mod" class="bare">https://stackoverflow.com/questions/3218320/what-is-the-difference-between-module-init-and-init-module-in-a-linux-kernel-mod</a></p>
</div>
</div>
<div class="sect3">
<h4 id="floating-point-in-kernel-modules"><a class="anchor" href="#floating-point-in-kernel-modules"></a><a class="link" href="#floating-point-in-kernel-modules">17.5.6. Floating point in kernel modules</a></h4>
<div class="paragraph">
<p>It is generally hard / impossible to use floating point operations in the kernel. TODO understand details.</p>
</div>
<div class="paragraph">
<p>A quick (x86-only for now because lazy) example is shown at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/float.c">kernel_modules/float.c</a></p>
</div>
<div class="paragraph">
<p>Usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod float.ko myfloat=1 enable_fpu=1</pre>
</div>
</div>
<div class="paragraph">
<p>We have to call: <code>kernel_fpu_begin()</code> before starting FPU operations, and <code>kernel_fpu_end()</code> when we are done. This particular example however did not blow up without it at lkmc 7f917af66b17373505f6c21d75af9331d624b3a9 + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod float.ko myfloat=1 enable_fpu=0</pre>
</div>
</div>
<div class="paragraph">
<p>The v5.1 documentation under <a href="https://github.com/cirosantilli/linux/blob/v5.1/arch/x86/include/asm/fpu/api.h#L15">arch/x86/include/asm/fpu/api.h</a> reads:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> * Use kernel_fpu_begin/end() if you intend to use FPU in kernel context. It
 * disables preemption so be careful if you intend to use it for long periods
 * of time.</pre>
</div>
</div>
<div class="paragraph">
<p>The example sets in the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/Makefile">kernel_modules/Makefile</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CFLAGS_REMOVE_float.o += -mno-sse -mno-sse2</pre>
</div>
</div>
<div class="paragraph">
<p>to avoid:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>error: SSE register return with SSE disabled</pre>
</div>
</div>
<div class="paragraph">
<p>We found those flags with <code>./build-modules --verbose</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/13886338/use-of-floating-point-in-the-linux-kernel" class="bare">https://stackoverflow.com/questions/13886338/use-of-floating-point-in-the-linux-kernel</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/15883947/why-am-i-able-to-perform-floating-point-operations-inside-a-linux-kernel-module/47056242" class="bare">https://stackoverflow.com/questions/15883947/why-am-i-able-to-perform-floating-point-operations-inside-a-linux-kernel-module/47056242</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/1556142/sse-register-return-with-sse-disabled" class="bare">https://stackoverflow.com/questions/1556142/sse-register-return-with-sse-disabled</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel-panic-and-oops"><a class="anchor" href="#kernel-panic-and-oops"></a><a class="link" href="#kernel-panic-and-oops">17.6. Kernel panic and oops</a></h3>
<div class="paragraph">
<p>To test out kernel panics and oops in controlled circumstances, try out the modules:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod panic.ko
insmod oops.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/panic.c">kernel_modules/panic.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/oops.c">kernel_modules/oops.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>A panic can also be generated with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo c &gt; /proc/sysrq-trigger</pre>
</div>
</div>
<div class="paragraph">
<p>Panic vs oops: <a href="https://unix.stackexchange.com/questions/91854/whats-the-difference-between-a-kernel-oops-and-a-kernel-panic" class="bare">https://unix.stackexchange.com/questions/91854/whats-the-difference-between-a-kernel-oops-and-a-kernel-panic</a></p>
</div>
<div class="paragraph">
<p>How to generate them:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/66197/how-to-cause-kernel-panic-with-a-single-command" class="bare">https://unix.stackexchange.com/questions/66197/how-to-cause-kernel-panic-with-a-single-command</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/23484147/generate-kernel-oops-or-crash-in-the-code" class="bare">https://stackoverflow.com/questions/23484147/generate-kernel-oops-or-crash-in-the-code</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>When a panic happens, <a href="#linux-kernel-magic-keys"><code>Shift-PgUp</code></a> does not work as it normally does, and it is hard to get the logs if on are on <a href="#qemu-graphic-mode">QEMU graphic mode</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/848412/scrolling-up-the-failed-screen-with-kernel-panic" class="bare">https://superuser.com/questions/848412/scrolling-up-the-failed-screen-with-kernel-panic</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/269228/write-qemu-booting-virtual-machine-output-to-a-file" class="bare">https://superuser.com/questions/269228/write-qemu-booting-virtual-machine-output-to-a-file</a></p>
</li>
<li>
<p><a href="http://www.reactos.org/wiki/QEMU#Redirect_to_a_file" class="bare">http://www.reactos.org/wiki/QEMU#Redirect_to_a_file</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="kernel-panic"><a class="anchor" href="#kernel-panic"></a><a class="link" href="#kernel-panic">17.6.1. Kernel panic</a></h4>
<div class="paragraph">
<p>On panic, the kernel dies, and so does our terminal.</p>
</div>
<div class="paragraph">
<p>The panic trace looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>panic: loading out-of-tree module taints kernel.
panic myinit
Kernel panic - not syncing: hello panic
CPU: 0 PID: 53 Comm: insmod Tainted: G           O     4.16.0 #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
Call Trace:
 dump_stack+0x7d/0xba
 ? 0xffffffffc0000000
 panic+0xda/0x213
 ? printk+0x43/0x4b
 ? 0xffffffffc0000000
 myinit+0x1d/0x20 [panic]
 do_one_initcall+0x3e/0x170
 do_init_module+0x5b/0x210
 load_module+0x2035/0x29d0
 ? kernel_read_file+0x7d/0x140
 ? SyS_finit_module+0xa8/0xb0
 SyS_finit_module+0xa8/0xb0
 do_syscall_64+0x6f/0x310
 ? trace_hardirqs_off_thunk+0x1a/0x32
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x7ffff7b36206
RSP: 002b:00007fffffffeb78 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
RAX: ffffffffffffffda RBX: 000000000000005c RCX: 00007ffff7b36206
RDX: 0000000000000000 RSI: 000000000069e010 RDI: 0000000000000003
RBP: 000000000069e010 R08: 00007ffff7ddd320 R09: 0000000000000000
R10: 00007ffff7ddd320 R11: 0000000000000206 R12: 0000000000000003
R13: 00007fffffffef4a R14: 0000000000000000 R15: 0000000000000000
Kernel Offset: disabled
---[ end Kernel panic - not syncing: hello panic</pre>
</div>
</div>
<div class="paragraph">
<p>Notice how our panic message <code>hello panic</code> is visible at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Kernel panic - not syncing: hello panic</pre>
</div>
</div>
<div class="sect4">
<h5 id="kernel-module-stack-trace-to-source-line"><a class="anchor" href="#kernel-module-stack-trace-to-source-line"></a><a class="link" href="#kernel-module-stack-trace-to-source-line">17.6.1.1. Kernel module stack trace to source line</a></h5>
<div class="paragraph">
<p>The log shows which module each symbol belongs to if any, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>myinit+0x1d/0x20 [panic]</pre>
</div>
</div>
<div class="paragraph">
<p>says that the function <code>myinit</code> is in the module <code>panic</code>.</p>
</div>
<div class="paragraph">
<p>To find the line that panicked, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb</pre>
</div>
</div>
<div class="paragraph">
<p>and then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info line *(myinit+0x1d)</pre>
</div>
</div>
<div class="paragraph">
<p>which gives us the correct line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Line 7 of "/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c" starts at address 0xbf00001c &lt;myinit+28&gt; and ends at 0xbf00002c &lt;myexit&gt;.</pre>
</div>
</div>
<div class="paragraph">
<p>as explained at: <a href="https://stackoverflow.com/questions/8545931/using-gdb-to-convert-addresses-to-lines/27576029#27576029" class="bare">https://stackoverflow.com/questions/8545931/using-gdb-to-convert-addresses-to-lines/27576029#27576029</a></p>
</div>
<div class="paragraph">
<p>The exact same thing can be done post mortem with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain gdb -- \
  -batch \
  -ex 'info line *(myinit+0x1d)' \
  "$(./getvar kernel_modules_build_subdir)/panic.ko" \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Related:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/6151538/addr2line-on-kernel-module" class="bare">https://stackoverflow.com/questions/6151538/addr2line-on-kernel-module</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/13468286/how-to-read-understand-analyze-and-debug-a-linux-kernel-panic" class="bare">https://stackoverflow.com/questions/13468286/how-to-read-understand-analyze-and-debug-a-linux-kernel-panic</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="bug-on"><a class="anchor" href="#bug-on"></a><a class="link" href="#bug-on">17.6.1.2. BUG_ON</a></h5>
<div class="paragraph">
<p>Basically just calls <code>panic("BUG!")</code> for most archs.</p>
</div>
</div>
<div class="sect4">
<h5 id="exit-emulator-on-panic"><a class="anchor" href="#exit-emulator-on-panic"></a><a class="link" href="#exit-emulator-on-panic">17.6.1.3. Exit emulator on panic</a></h5>
<div class="paragraph">
<p>For testing purposes, it is very useful to quit the emulator automatically with exit status non zero in case of kernel panic, instead of just hanging forever.</p>
</div>
<div class="sect5">
<h6 id="exit-qemu-on-panic"><a class="anchor" href="#exit-qemu-on-panic"></a><a class="link" href="#exit-qemu-on-panic">17.6.1.3.1. Exit QEMU on panic</a></h6>
<div class="paragraph">
<p>Enabled by default with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>panic=-1</code> command line option which reboots the kernel immediately on panic, see: <a href="#reboot-on-panic">Section 17.6.1.4, &#8220;Reboot on panic&#8221;</a></p>
</li>
<li>
<p>QEMU <code>-no-reboot</code>, which makes QEMU exit when the guest tries to reboot</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Also asked at <a href="https://unix.stackexchange.com/questions/443017/can-i-make-qemu-exit-with-failure-on-kernel-panic" class="bare">https://unix.stackexchange.com/questions/443017/can-i-make-qemu-exit-with-failure-on-kernel-panic</a> which also mentions the x86_64 <code>-device pvpanic</code>, but I don&#8217;t see much advantage to it.</p>
</div>
<div class="paragraph">
<p>TODO neither method exits with exit status different from 0, so for now we are just grepping the logs for panic messages, which sucks.</p>
</div>
<div class="paragraph">
<p>One possibility that gets close would be to use <a href="#gdb">GDB step debug</a> to break at the <code>panic</code> function, and then send a <a href="#qemu-monitor-from-gdb">QEMU monitor from GDB</a> <code>quit</code> command if that happens, but I don&#8217;t see a way to exit with non-zero status to indicate error.</p>
</div>
</div>
<div class="sect5">
<h6 id="exit-gem5-on-panic"><a class="anchor" href="#exit-gem5-on-panic"></a><a class="link" href="#exit-gem5-on-panic">17.6.1.3.2. Exit gem5 on panic</a></h6>
<div class="paragraph">
<p>gem5 9048ef0ffbf21bedb803b785fb68f83e95c04db8 (January 2019) can detect panics automatically if the option <code>system.panic_on_panic</code> is on.</p>
</div>
<div class="paragraph">
<p>It parses kernel symbols and detecting when the PC reaches the address of the <code>panic</code> function. gem5 then prints to stdout:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Kernel panic in simulated kernel</pre>
</div>
</div>
<div class="paragraph">
<p>and exits with status -6.</p>
</div>
<div class="paragraph">
<p>At gem5 ff52563a214c71fcd1e21e9f00ad839612032e3b (July 2018) behaviour was different, and just exited 0: <a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15870.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15870.html</a> TODO find fixing commit.</p>
</div>
<div class="paragraph">
<p>We enable the <code>system.panic_on_panic</code> option by default on <code>arm</code> and <code>aarch64</code>, which makes gem5 exit immediately in case of panic, which is awesome!</p>
</div>
<div class="paragraph">
<p>If we don&#8217;t set <code>system.panic_on_panic</code>, then gem5 just hangs on an infinite guest loop.</p>
</div>
<div class="paragraph">
<p>TODO: why doesn&#8217;t gem5 x86 ff52563a214c71fcd1e21e9f00ad839612032e3b support <code>system.panic_on_panic</code> as well? Trying to set <code>system.panic_on_panic</code> there fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>tried to set or access non-existentobject parameter: panic_on_panic</pre>
</div>
</div>
<div class="paragraph">
<p>However, at that commit panic on x86 makes gem5 crash with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>panic: i8042 "System reset" command not implemented.</pre>
</div>
</div>
<div class="paragraph">
<p>which is a good side effect of an unimplemented hardware feature, since the simulation actually stops.</p>
</div>
<div class="paragraph">
<p>The implementation of panic detection happens at: <a href="https://github.com/gem5/gem5/blob/1da285dfcc31b904afc27e440544d006aae25b38/src/arch/arm/linux/system.cc#L73" class="bare">https://github.com/gem5/gem5/blob/1da285dfcc31b904afc27e440544d006aae25b38/src/arch/arm/linux/system.cc#L73</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>        kernelPanicEvent = addKernelFuncEventOrPanic&lt;Linux::KernelPanicEvent&gt;(
            "panic", "Kernel panic in simulated kernel", dmesg_output);</pre>
</div>
</div>
<div class="paragraph">
<p>Here we see that the symbol <code>"panic"</code> for the <code>panic()</code> function is the one being tracked.</p>
</div>
<div class="paragraph">
<p>Related thread: <a href="https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully" class="bare">https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="reboot-on-panic"><a class="anchor" href="#reboot-on-panic"></a><a class="link" href="#reboot-on-panic">17.6.1.4. Reboot on panic</a></h5>
<div class="paragraph">
<p>Make the kernel reboot after n seconds after panic:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/kernel/panic</pre>
</div>
</div>
<div class="paragraph">
<p>Can also be controlled with the <code>panic=</code> kernel boot parameter.</p>
</div>
<div class="paragraph">
<p><code>0</code> to disable, <code>-1</code> to reboot immediately.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.17/Documentation/admin-guide/kernel-parameters.txt#L2931" class="bare">https://github.com/torvalds/linux/blob/v4.17/Documentation/admin-guide/kernel-parameters.txt#L2931</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/29567/how-to-configure-the-linux-kernel-to-reboot-on-panic/29569#29569" class="bare">https://unix.stackexchange.com/questions/29567/how-to-configure-the-linux-kernel-to-reboot-on-panic/29569#29569</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="panic-trace-show-addresses-instead-of-symbols"><a class="anchor" href="#panic-trace-show-addresses-instead-of-symbols"></a><a class="link" href="#panic-trace-show-addresses-instead-of-symbols">17.6.1.5. Panic trace show addresses instead of symbols</a></h5>
<div class="paragraph">
<p>If <code>CONFIG_KALLSYMS=n</code>, then addresses are shown on traces instead of symbol plus offset.</p>
</div>
<div class="paragraph">
<p>In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'insmod dump_stack.ko' --gdb-wait --tmux-args dump_stack</pre>
</div>
</div>
<div class="paragraph">
<p>shows that traces are printed at <code>arch/x86/kernel/dumpstack.c</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static void printk_stack_address(unsigned long address, int reliable,
                 char *log_lvl)
{
    touch_nmi_watchdog();
    printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address);
}</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>%pB</code> is documented at <code>Documentation/core-api/printk-formats.rst</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>If KALLSYMS are disabled then the symbol address is printed instead.</pre>
</div>
</div>
<div class="paragraph">
<p>I wasn&#8217;t able do disable <code>CONFIG_KALLSYMS</code> to test this this out however, it is being selected by some other option? But I then used <code>make menuconfig</code> to see which options select it, and they were all off&#8230;&#8203;</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="oops"><a class="anchor" href="#oops"></a><a class="link" href="#oops">17.6.2. Kernel oops</a></h4>
<div class="paragraph">
<p>On oops, the shell still lives after.</p>
</div>
<div class="paragraph">
<p>However we:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>leave the normal control flow, and <code>oops after</code> never gets printed: an interrupt is serviced</p>
</li>
<li>
<p>cannot <code>rmmod oops</code> afterwards</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It is possible to make <code>oops</code> lead to panics always with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/kernel/panic_on_oops
insmod oops.ko</pre>
</div>
</div>
<div class="paragraph">
<p>An oops stack trace looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: myinit+0x18/0x30 [oops]
PGD dccf067 P4D dccf067 PUD dcc1067 PMD 0
Oops: 0002 [#1] SMP NOPTI
Modules linked in: oops(O+)
CPU: 0 PID: 53 Comm: insmod Tainted: G           O     4.16.0 #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
RIP: 0010:myinit+0x18/0x30 [oops]
RSP: 0018:ffffc900000d3cb0 EFLAGS: 00000282
RAX: 000000000000000b RBX: ffffffffc0000000 RCX: ffffffff81e3e3a8
RDX: 0000000000000001 RSI: 0000000000000086 RDI: ffffffffc0001033
RBP: ffffc900000d3e30 R08: 69796d2073706f6f R09: 000000000000013b
R10: ffffea0000373280 R11: ffffffff822d8b2d R12: 0000000000000000
R13: ffffffffc0002050 R14: ffffffffc0002000 R15: ffff88000dc934c8
FS:  00007ffff7ff66a0(0000) GS:ffff88000fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000000dcd2000 CR4: 00000000000006f0
Call Trace:
 do_one_initcall+0x3e/0x170
 do_init_module+0x5b/0x210
 load_module+0x2035/0x29d0
 ? SyS_finit_module+0xa8/0xb0
 SyS_finit_module+0xa8/0xb0
 do_syscall_64+0x6f/0x310
 ? trace_hardirqs_off_thunk+0x1a/0x32
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x7ffff7b36206
RSP: 002b:00007fffffffeb78 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
RAX: ffffffffffffffda RBX: 000000000000005c RCX: 00007ffff7b36206
RDX: 0000000000000000 RSI: 000000000069e010 RDI: 0000000000000003
RBP: 000000000069e010 R08: 00007ffff7ddd320 R09: 0000000000000000
R10: 00007ffff7ddd320 R11: 0000000000000206 R12: 0000000000000003
R13: 00007fffffffef4b R14: 0000000000000000 R15: 0000000000000000
Code: &lt;c7&gt; 04 25 00 00 00 00 00 00 00 00 e8 b2 33 09 c1 31 c0 c3 0f 1f 44
RIP: myinit+0x18/0x30 [oops] RSP: ffffc900000d3cb0
CR2: 0000000000000000
---[ end trace 3cdb4e9d9842b503 ]---</pre>
</div>
</div>
<div class="paragraph">
<p>To find the line that oopsed, look at the <code>RIP</code> register:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>RIP: 0010:myinit+0x18/0x30 [oops]</pre>
</div>
</div>
<div class="paragraph">
<p>and then on GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb</pre>
</div>
</div>
<div class="paragraph">
<p>run</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info line *(myinit+0x18)</pre>
</div>
</div>
<div class="paragraph">
<p>which gives us the correct line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Line 7 of "/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c" starts at address 0xbf00001c &lt;myinit+28&gt; and ends at 0xbf00002c &lt;myexit&gt;.</pre>
</div>
</div>
<div class="paragraph">
<p>This-did not work on <code>arm</code> due to <a href="#gdb-step-debug-kernel-module-arm">GDB step debug kernel module insmodded by init on ARM</a> so we need to either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gdb-module-init">GDB module_init</a></p>
</li>
<li>
<p><a href="#kernel-module-stack-trace-to-source-line">Kernel module stack trace to source line</a> post-mortem method</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="dump-stack"><a class="anchor" href="#dump-stack"></a><a class="link" href="#dump-stack">17.6.3. dump_stack</a></h4>
<div class="paragraph">
<p>The <code>dump_stack</code> function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod dump_stack.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/dump_stack.c">kernel_modules/dump_stack.c</a></p>
</div>
</div>
<div class="sect3">
<h4 id="warn-on"><a class="anchor" href="#warn-on"></a><a class="link" href="#warn-on">17.6.4. WARN_ON</a></h4>
<div class="paragraph">
<p>The <code>WARN_ON</code> macro basically just calls <a href="#dump-stack">dump_stack</a>.</p>
</div>
<div class="paragraph">
<p>One extra side effect is that we can make it also panic with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/kernel/panic_on_warn
insmod warn_on.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/warn_on.c">kernel_modules/warn_on.c</a></p>
</div>
<div class="paragraph">
<p>Can also be activated with the <code>panic_on_warn</code> boot parameter.</p>
</div>
</div>
<div class="sect3">
<h4 id="not-syncing-vfs"><a class="anchor" href="#not-syncing-vfs"></a><a class="link" href="#not-syncing-vfs">17.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)</a></h4>
<div class="paragraph">
<p>Let&#8217;s learn how to diagnose problems with the root filesystem not being found. TODO add a sample panic error message for each error type:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477" class="bare">https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is the diagnosis procedure.</p>
</div>
<div class="paragraph">
<p>First, if we remove the following options from the our kernel build:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_PCI=y</pre>
</div>
</div>
<div class="paragraph">
<p>we get a message like this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;4&gt;[    0.541708] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
&lt;4&gt;[    0.542035] Please append a correct "root=" boot option; here are the available partitions:
&lt;0&gt;[    0.542562] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)</pre>
</div>
</div>
<div class="paragraph">
<p>From the message, we notice that the kernel sees a disk of some sort (vda means a virtio disk), but it could not open it.</p>
</div>
<div class="paragraph">
<p>This means that the kernel cannot properly read any bytes from the disk.</p>
</div>
<div class="paragraph">
<p>And afterwards, it has an useless message <code>here are the available partitions:</code>, but of course we have no available partitions, the list is empty, because the kernel cannot even read bytes from the disk, so it definitely cannot understand its filesystems.</p>
</div>
<div class="paragraph">
<p>This can indicate basically two things:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>on real hardware, it could mean that the hardware is broken. Kind of hard on emulators ;-)</p>
</li>
<li>
<p>you didn&#8217;t configure the kernel with the option that enables it to read from that kind of disk.</p>
<div class="paragraph">
<p>In our case, disks are virtio devices that QEMU exposes to the guest kernel. This is why removing the options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_PCI=y</pre>
</div>
</div>
<div class="paragraph">
<p>led to this error.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Now, let&#8217;s restore the previously removed virtio options, and instead remove:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_EXT4_FS=y</pre>
</div>
</div>
<div class="paragraph">
<p>This time, the kernel will be able to read bytes from the device. But it won&#8217;t be able to read files from the filesystem, because our filesystem is in ext4 format.</p>
</div>
<div class="paragraph">
<p>Therefore, this time the error message looks like this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;4&gt;[    0.585296] List of all partitions:
&lt;4&gt;[    0.585913] fe00          524288 vda
&lt;4&gt;[    0.586123]  driver: virtio_blk
&lt;4&gt;[    0.586471] No filesystem could mount root, tried:
&lt;4&gt;[    0.586497]  squashfs
&lt;4&gt;[    0.586724]
&lt;0&gt;[    0.587360] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)</pre>
</div>
</div>
<div class="paragraph">
<p>In this case, we see that the kernel did manage to read from the <code>vda</code> disk! It even told us how: by using the <code>driver: virtio_blk</code>.</p>
</div>
<div class="paragraph">
<p>However, it then went through the list of all filesystem types it knows how to read files from, in our case just <code>squashf</code>, and none of those worked, because our partition is an ext4 partition.</p>
</div>
<div class="paragraph">
<p>Finally, the last possible error is that we simply passed the wrong <code>root=</code> <a href="#kernel-command-line-parameters">kernel CLI option</a>. For example, if we hack our command to pass:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>root=/dev/vda2</pre>
</div>
</div>
<div class="paragraph">
<p>which does not even exist since <code>/dev/vda</code> is a raw non-partitioned ext4 image, then boot fails with a message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;4&gt;[    0.608475] Please append a correct "root=" boot option; here are the available partitions:
&lt;4&gt;[    0.609563] fe00          524288 vda
&lt;4&gt;[    0.609723]  driver: virtio_blk
&lt;0&gt;[    0.610433] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,2)</pre>
</div>
</div>
<div class="paragraph">
<p>This one is easy, because the kernel tells us clearly which partitions it would have been able to understand. In our case <code>/dev/vda</code>.</p>
</div>
<div class="paragraph">
<p>Once all those problems are solved, in the working setup, we finally see something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    0.636129] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
&lt;6&gt;[    0.636700] VFS: Mounted root (ext4 filesystem) on device 254:0.</pre>
</div>
</div>
<div class="paragraph">
<p>Tested on LKMC 863a373a30cd3c7982e3e453c4153f85133b17a9, Linux kernel 5.4.3.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477" class="bare">https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/414655/not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/603197#603197" class="bare">https://unix.stackexchange.com/questions/414655/not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/603197#603197</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/63277677/i-meet-a-problem-when-i-encountered-in-the-fs-mode-of-running-gem5/63278487#63278487" class="bare">https://stackoverflow.com/questions/63277677/i-meet-a-problem-when-i-encountered-in-the-fs-mode-of-running-gem5/63278487#63278487</a> summary only</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pseudo-filesystems"><a class="anchor" href="#pseudo-filesystems"></a><a class="link" href="#pseudo-filesystems">17.7. Pseudo filesystems</a></h3>
<div class="paragraph">
<p>Pseudo filesystems are filesystems that don&#8217;t represent actual files in a hard disk, but rather allow us to do special operations on filesystem-related system calls.</p>
</div>
<div class="paragraph">
<p>What each pseudo-file does for each related system call does is defined by its <a href="#file-operations">File operations</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/1198292/what-is-a-pseudo-file-system-in-linux" class="bare">https://superuser.com/questions/1198292/what-is-a-pseudo-file-system-in-linux</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Synthetic_file_system" class="bare">https://en.wikipedia.org/wiki/Synthetic_file_system</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="debugfs"><a class="anchor" href="#debugfs"></a><a class="link" href="#debugfs">17.7.1. debugfs</a></h4>
<div class="paragraph">
<p>Debugfs is the simplest pseudo filesystem to play around with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./debugfs.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/debugfs.c">kernel_modules/debugfs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/debugfs.sh">rootfs_overlay/lkmc/debugfs.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Debugfs is made specifically to help test kernel stuff. Just mount, set <a href="#file-operations">File operations</a>, and we are done.</p>
</div>
<div class="paragraph">
<p>For this reason, it is the filesystem that we use whenever possible in our tests.</p>
</div>
<div class="paragraph">
<p><code>debugfs.sh</code> explicitly mounts a debugfs at a custom location, but the most common mount point is <code>/sys/kernel/debug</code>.</p>
</div>
<div class="paragraph">
<p>This mount not done automatically by the kernel however: we, like most distros, do it from userland with our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/fstab">fstab</a>.</p>
</div>
<div class="paragraph">
<p>Debugfs support requires the kernel to be compiled with <code>CONFIG_DEBUG_FS=y</code>.</p>
</div>
<div class="paragraph">
<p>Only the more basic file operations can be implemented in debugfs, e.g. <code>mmap</code> never gets called:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://patchwork.kernel.org/patch/9252557/" class="bare">https://patchwork.kernel.org/patch/9252557/</a></p>
</li>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.9/fs/debugfs/file.c#L212" class="bare">https://github.com/torvalds/linux/blob/v4.9/fs/debugfs/file.c#L212</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://github.com/chadversary/debugfs-tutorial" class="bare">https://github.com/chadversary/debugfs-tutorial</a></p>
</div>
</div>
<div class="sect3">
<h4 id="procfs"><a class="anchor" href="#procfs"></a><a class="link" href="#procfs">17.7.2. procfs</a></h4>
<div class="paragraph">
<p>Procfs is just another fops entry point:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./procfs.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Procfs is a little less convenient than <a href="#debugfs">debugfs</a>, but is more used in serious applications.</p>
</div>
<div class="paragraph">
<p>Procfs can run all system calls, including ones that debugfs can&#8217;t, e.g. <a href="#mmap">mmap</a>.</p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/procfs.c">kernel_modules/procfs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/procfs.sh">rootfs_overlay/lkmc/procfs.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/619955/how-does-proc-work/1442571#1442571" class="bare">https://superuser.com/questions/619955/how-does-proc-work/1442571#1442571</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/8516021/proc-create-example-for-kernel-module/18924359#18924359" class="bare">https://stackoverflow.com/questions/8516021/proc-create-example-for-kernel-module/18924359#18924359</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="proc-version"><a class="anchor" href="#proc-version"></a><a class="link" href="#proc-version">17.7.2.1. /proc/version</a></h5>
<div class="paragraph">
<p>Its data is shared with <code>uname()</code>, which is a <a href="#posix">POSIX C</a> function and has a Linux syscall to back it up.</p>
</div>
<div class="paragraph">
<p>Where the data comes from and how to modify it:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/136959/where-does-uname-get-its-information-from/485962#485962" class="bare">https://unix.stackexchange.com/questions/136959/where-does-uname-get-its-information-from/485962#485962</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/23424174/how-to-customize-or-remove-extra-linux-kernel-version-details-shown-at-boot" class="bare">https://stackoverflow.com/questions/23424174/how-to-customize-or-remove-extra-linux-kernel-version-details-shown-at-boot</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In this repo, leaking host information, and to make builds more reproducible, we are setting:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>user and date to dummy values with <code>KBUILD_BUILD_USER</code> and <code>KBUILD_BUILD_TIMESTAMP</code></p>
</li>
<li>
<p>hostname to the kernel git commit with <code>KBUILD_BUILD_HOST</code> and <code>KBUILD_BUILD_VERSION</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>A sample result is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Linux version 4.19.0-dirty (lkmc@84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d) (gcc version 6.4.0 (Buildroot 2018.05-00002-gbc60382b8f)) #1 SMP Thu Jan 1 00:00:00 UTC 1970</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="sysfs"><a class="anchor" href="#sysfs"></a><a class="link" href="#sysfs">17.7.3. sysfs</a></h4>
<div class="paragraph">
<p>Sysfs is more restricted than <a href="#procfs">procfs</a>, as it does not take an arbitrary <code>file_operations</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./sysfs.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/sysfs.c">kernel_modules/sysfs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/sysfs.sh">rootfs_overlay/lkmc/sysfs.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Vs procfs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/4884/what-is-the-difference-between-procfs-and-sysfs/382315#382315" class="bare">https://unix.stackexchange.com/questions/4884/what-is-the-difference-between-procfs-and-sysfs/382315#382315</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/37237835/how-to-attach-file-operations-to-sysfs-attribute-in-platform-driver" class="bare">https://stackoverflow.com/questions/37237835/how-to-attach-file-operations-to-sysfs-attribute-in-platform-driver</a></p>
</li>
<li>
<p><a href="https://serverfault.com/questions/65261/linux-proc-sys-kernel-vs-sys-kernel" class="bare">https://serverfault.com/questions/65261/linux-proc-sys-kernel-vs-sys-kernel</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>You basically can only do <code>open</code>, <code>close</code>, <code>read</code>, <code>write</code>, and <code>lseek</code> on sysfs files.</p>
</div>
<div class="paragraph">
<p>It is similar to a <a href="#seq-file">seq_file</a> file operation, except that write is also implemented.</p>
</div>
<div class="paragraph">
<p>TODO: what are those <code>kobject</code> structs? Make a more complex example that shows what they can do.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/t3rm1n4l/kern-dev-tutorial/blob/1f036ef40fc4378f5c8d2842e55bcea7c6f8894a/05-sysfs/sysfs.c" class="bare">https://github.com/t3rm1n4l/kern-dev-tutorial/blob/1f036ef40fc4378f5c8d2842e55bcea7c6f8894a/05-sysfs/sysfs.c</a></p>
</li>
<li>
<p><a href="https://www.kernel.org/doc/Documentation/kobject.txt" class="bare">https://www.kernel.org/doc/Documentation/kobject.txt</a></p>
</li>
<li>
<p><a href="https://www.quora.com/What-are-kernel-objects-Kobj" class="bare">https://www.quora.com/What-are-kernel-objects-Kobj</a></p>
</li>
<li>
<p><a href="http://www.makelinux.net/ldd3/chp-14-sect-1" class="bare">http://www.makelinux.net/ldd3/chp-14-sect-1</a></p>
</li>
<li>
<p><a href="https://www.win.tue.nl/~aeb/linux/lk/lk-13.html" class="bare">https://www.win.tue.nl/~aeb/linux/lk/lk-13.html</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="character-devices"><a class="anchor" href="#character-devices"></a><a class="link" href="#character-devices">17.7.4. Character devices</a></h4>
<div class="paragraph">
<p>Character devices can have arbitrary <a href="#file-operations">File operations</a> associated to them:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./character_device.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/character_device.sh">rootfs_overlay/lkmc/character_device.sh</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/mknoddev.sh">rootfs_overlay/lkmc/mknoddev.sh</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/character_device.c">kernel_modules/character_device.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Unlike <a href="#procfs">procfs</a> entires, character device files are created with userland <code>mknod</code> or <code>mknodat</code> syscalls:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mknod &lt;/dev/path_to_dev&gt; c &lt;major&gt; &lt;minor&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>Intuitively, for physical devices like keyboards, the major number maps to which driver, and the minor number maps to which device it is.</p>
</div>
<div class="paragraph">
<p>A single driver can drive multiple compatible devices.</p>
</div>
<div class="paragraph">
<p>The major and minor numbers can be observed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls -l /dev/urandom</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>crw-rw-rw-    1 root     root        1,   9 Jun 29 05:45 /dev/urandom</pre>
</div>
</div>
<div class="paragraph">
<p>which means:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>c</code> (first letter): this is a character device. Would be <code>b</code> for a block device.</p>
</li>
<li>
<p><code>1,   9</code>: the major number is <code>1</code>, and the minor <code>9</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To avoid device number conflicts when registering the driver we:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ask the kernel to allocate a free major number for us with: <code>register_chrdev(0</code></p>
</li>
<li>
<p>find ouf which number was assigned by grepping <code>/proc/devices</code> for the kernel module name</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://unix.stackexchange.com/questions/37829/understanding-character-device-or-character-special-files/371758#371758" class="bare">https://unix.stackexchange.com/questions/37829/understanding-character-device-or-character-special-files/371758#371758</a></p>
</div>
<div class="sect4">
<h5 id="automatically-create-character-device-file-on-insmod"><a class="anchor" href="#automatically-create-character-device-file-on-insmod"></a><a class="link" href="#automatically-create-character-device-file-on-insmod">17.7.4.1. Automatically create character device file on insmod</a></h5>
<div class="paragraph">
<p>And also destroy it on <code>rmmod</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./character_device_create.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/character_device_create.c">kernel_modules/character_device_create.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/character_device_create.sh">rootfs_overlay/lkmc/character_device_create.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/5970595/how-to-create-a-device-node-from-the-init-module-code-of-a-linux-kernel-module/45531867#45531867" class="bare">https://stackoverflow.com/questions/5970595/how-to-create-a-device-node-from-the-init-module-code-of-a-linux-kernel-module/45531867#45531867</a></p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pseudo-files"><a class="anchor" href="#pseudo-files"></a><a class="link" href="#pseudo-files">17.8. Pseudo files</a></h3>
<div class="sect3">
<h4 id="file-operations"><a class="anchor" href="#file-operations"></a><a class="link" href="#file-operations">17.8.1. File operations</a></h4>
<div class="paragraph">
<p>File operations are the main method of userland driver communication.</p>
</div>
<div class="paragraph">
<p><code>struct file_operations</code> determines what the kernel will do on filesystem system calls of <a href="#pseudo-filesystems">Pseudo filesystems</a>.</p>
</div>
<div class="paragraph">
<p>This example illustrates the most basic system calls: <code>open</code>, <code>read</code>, <code>write</code>, <code>close</code> and <code>lseek</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./fops.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/fops.c">kernel_modules/fops.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/fops.sh">rootfs_overlay/lkmc/fops.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then give this a try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sh -x ./fops.sh</pre>
</div>
</div>
<div class="paragraph">
<p>We have put printks on each fop, so this allows you to see which system calls are being made for each command.</p>
</div>
<div class="paragraph">
<p>No, there no official documentation: <a href="https://stackoverflow.com/questions/15213932/what-are-the-struct-file-operations-arguments" class="bare">https://stackoverflow.com/questions/15213932/what-are-the-struct-file-operations-arguments</a></p>
</div>
</div>
<div class="sect3">
<h4 id="seq-file"><a class="anchor" href="#seq-file"></a><a class="link" href="#seq-file">17.8.2. seq_file</a></h4>
<div class="paragraph">
<p>Writing trivial read <a href="#file-operations">File operations</a> is repetitive and error prone. The <code>seq_file</code> API makes the process much easier for those trivial cases:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./seq_file.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/seq_file.c">kernel_modules/seq_file.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/seq_file.sh">rootfs_overlay/lkmc/seq_file.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In this example we create a debugfs file that behaves just like a file that contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0
1
2</pre>
</div>
</div>
<div class="paragraph">
<p>However, we only store a single integer in memory and calculate the file on the fly in an iterator fashion.</p>
</div>
<div class="paragraph">
<p><code>seq_file</code> does not provide <code>write</code>: <a href="https://stackoverflow.com/questions/30710517/how-to-implement-a-writable-proc-file-by-using-seq-file-in-a-driver-module" class="bare">https://stackoverflow.com/questions/30710517/how-to-implement-a-writable-proc-file-by-using-seq-file-in-a-driver-module</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.17/Documentation/filesystems/seq_file.txt">Documentation/filesystems/seq_file.txt</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/25399112/how-to-use-a-seq-file-in-linux-modules" class="bare">https://stackoverflow.com/questions/25399112/how-to-use-a-seq-file-in-linux-modules</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="seq-file-single-open"><a class="anchor" href="#seq-file-single-open"></a><a class="link" href="#seq-file-single-open">17.8.2.1. seq_file single_open</a></h5>
<div class="paragraph">
<p>If you have the entire read output upfront, <code>single_open</code> is an even more convenient version of <a href="#seq-file">seq_file</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./seq_file.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/seq_file_single_open.c">kernel_modules/seq_file_single_open.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/seq_file_single_open.sh">rootfs_overlay/lkmc/seq_file_single_open.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This example produces a debugfs file that behaves like a file that contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ab
cd</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="poll"><a class="anchor" href="#poll"></a><a class="link" href="#poll">17.8.3. poll</a></h4>
<div class="paragraph">
<p>The poll system call allows an user process to do a non-busy wait on a kernel event.</p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/poll.c">kernel_modules/poll.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/poll.sh">rootfs_overlay/lkmc/poll.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./poll.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: <code>jiffies</code> gets printed to stdout every second from userland, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>poll
&lt;6&gt;[    4.275305] poll
&lt;6&gt;[    4.275580] return POLLIN
revents = 1
POLLIN n=10 buf=4294893337
poll
&lt;6&gt;[    4.276627] poll
&lt;6&gt;[    4.276911] return 0
&lt;6&gt;[    5.271193] wake_up
&lt;6&gt;[    5.272326] poll
&lt;6&gt;[    5.273207] return POLLIN
revents = 1
POLLIN n=10 buf=4294893588
poll
&lt;6&gt;[    5.276367] poll
&lt;6&gt;[    5.276618] return 0
&lt;6&gt;[    6.275178] wake_up
&lt;6&gt;[    6.276370] poll
&lt;6&gt;[    6.277269] return POLLIN
revents = 1
POLLIN n=10 buf=4294893839</pre>
</div>
</div>
<div class="paragraph">
<p>Force the poll <a href="#file-operations"><code>file_operation</code></a> to return 0 to see what happens more clearly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./poll.sh pol0=1</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>poll
&lt;6&gt;[   85.674801] poll
&lt;6&gt;[   85.675788] return 0
&lt;6&gt;[   86.675182] wake_up
&lt;6&gt;[   86.676431] poll
&lt;6&gt;[   86.677373] return 0
&lt;6&gt;[   87.679198] wake_up
&lt;6&gt;[   87.680515] poll
&lt;6&gt;[   87.681564] return 0
&lt;6&gt;[   88.683198] wake_up</pre>
</div>
</div>
<div class="paragraph">
<p>From this we see that control is not returned to userland: the kernel just keeps calling the poll <code>file_operation</code> again and again.</p>
</div>
<div class="paragraph">
<p>Typically, we are waiting for some hardware to make some piece of data available available to the kernel.</p>
</div>
<div class="paragraph">
<p>The hardware notifies the kernel that the data is ready with an interrupt.</p>
</div>
<div class="paragraph">
<p>To simplify this example, we just fake the hardware interrupts with a <a href="#kthread">kthread</a> that sleeps for a second in an infinite loop.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/30035776/how-to-add-poll-function-to-the-kernel-module-code/44645336#44645336" class="bare">https://stackoverflow.com/questions/30035776/how-to-add-poll-function-to-the-kernel-module-code/44645336#44645336</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/30234496/why-do-we-need-to-call-poll-wait-in-poll/44645480#44645480" class="bare">https://stackoverflow.com/questions/30234496/why-do-we-need-to-call-poll-wait-in-poll/44645480#44645480</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="ioctl"><a class="anchor" href="#ioctl"></a><a class="link" href="#ioctl">17.8.4. ioctl</a></h4>
<div class="paragraph">
<p>The <code>ioctl</code> system call is the best way to pass an arbitrary number of parameters to the kernel in a single go:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./ioctl.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/ioctl.c">kernel_modules/ioctl.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/ioctl.h">lkmc/ioctl.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/kernel_modules/ioctl.c">userland/kernel_modules/ioctl.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/ioctl.sh">rootfs_overlay/lkmc/ioctl.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>ioctl</code> is one of the most important methods of communication with real device drivers, which often take several fields as input.</p>
</div>
<div class="paragraph">
<p><code>ioctl</code> takes as input:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>an integer <code>request</code> : it usually identifies what type of operation we want to do on this call</p>
</li>
<li>
<p>an untyped pointer to memory: can be anything, but is typically a pointer to a <code>struct</code></p>
<div class="paragraph">
<p>The type of the <code>struct</code> often depends on the <code>request</code> input</p>
</div>
<div class="paragraph">
<p>This <code>struct</code> is defined on a uapi-style C header that is used both to compile the kernel module and the userland executable.</p>
</div>
<div class="paragraph">
<p>The fields of this <code>struct</code> can be thought of as arbitrary input parameters.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>And the output is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>an integer return value. <code>man ioctl</code> documents:</p>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Usually, on success zero is returned. A few <code>ioctl()</code> requests use the return value as an output parameter and return a nonnegative value on success. On error, -1 is returned, and errno is set appropriately.</p>
</div>
</blockquote>
</div>
</li>
<li>
<p>the input pointer data may be overwritten to contain arbitrary output</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/2264384/how-do-i-use-ioctl-to-manipulate-my-kernel-module/44613896#44613896" class="bare">https://stackoverflow.com/questions/2264384/how-do-i-use-ioctl-to-manipulate-my-kernel-module/44613896#44613896</a></p>
</li>
<li>
<p><a href="https://askubuntu.com/questions/54239/problem-with-ioctl-in-a-simple-kernel-module/926675#926675" class="bare">https://askubuntu.com/questions/54239/problem-with-ioctl-in-a-simple-kernel-module/926675#926675</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="mmap"><a class="anchor" href="#mmap"></a><a class="link" href="#mmap">17.8.5. mmap</a></h4>
<div class="paragraph">
<p>The <code>mmap</code> system call allows us to share memory between user and kernel space without copying:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./mmap.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/mmap.c">kernel_modules/mmap.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/kernel_modules/mmap.c">userland/kernel_modules/mmap.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/mmap.sh">rootfs_overlay/lkmc/mmap.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In this example, we make a tiny 4 byte kernel buffer available to user-space, and we then modify it on userspace, and check that the kernel can see the modification.</p>
</div>
<div class="paragraph">
<p><code>mmap</code>, like most more complex <a href="#file-operations">File operations</a>, does not work with <a href="#debugfs">debugfs</a> as of 4.9, so we use a <a href="#procfs">procfs</a> file for it.</p>
</div>
<div class="paragraph">
<p>Example adapted from: <a href="https://coherentmusings.wordpress.com/2014/06/10/implementing-mmap-for-transferring-data-from-user-space-to-kernel-space/" class="bare">https://coherentmusings.wordpress.com/2014/06/10/implementing-mmap-for-transferring-data-from-user-space-to-kernel-space/</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/10760479/mmap-kernel-buffer-to-user-space/10770582#10770582" class="bare">https://stackoverflow.com/questions/10760479/mmap-kernel-buffer-to-user-space/10770582#10770582</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/1623008/allocating-memory-for-user-space-from-kernel-thread" class="bare">https://stackoverflow.com/questions/1623008/allocating-memory-for-user-space-from-kernel-thread</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/6967933/mmap-mapping-in-user-space-a-kernel-buffer-allocated-with-kmalloc" class="bare">https://stackoverflow.com/questions/6967933/mmap-mapping-in-user-space-a-kernel-buffer-allocated-with-kmalloc</a></p>
</li>
<li>
<p><a href="https://github.com/jeremytrimble/ezdma" class="bare">https://github.com/jeremytrimble/ezdma</a></p>
</li>
<li>
<p><a href="https://github.com/simonjhall/dma" class="bare">https://github.com/simonjhall/dma</a></p>
</li>
<li>
<p><a href="https://github.com/ikwzm/udmabuf" class="bare">https://github.com/ikwzm/udmabuf</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="anonymous-inode"><a class="anchor" href="#anonymous-inode"></a><a class="link" href="#anonymous-inode">17.8.6. Anonymous inode</a></h4>
<div class="paragraph">
<p>Anonymous inodes allow getting multiple file descriptors from a single filesystem entry, which reduces namespace pollution compared to creating multiple device files:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./anonymous_inode.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/anonymous_inode.c">kernel_modules/anonymous_inode.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/anonymous_inode.h">lkmc/anonymous_inode.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/kernel_modules/anonymous_inode.c">userland/kernel_modules/anonymous_inode.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/anonymous_inode.sh">rootfs_overlay/lkmc/anonymous_inode.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This example gets an anonymous inode via <a href="#ioctl">ioctl</a> from a debugfs entry by using <code>anon_inode_getfd</code>.</p>
</div>
<div class="paragraph">
<p>Reads to that inode return the sequence: <code>1</code>, <code>10</code>, <code>100</code>, &#8230;&#8203; <code>10000000</code>, <code>1</code>, <code>100</code>, &#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030" class="bare">https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030</a></p>
</div>
</div>
<div class="sect3">
<h4 id="netlink-sockets"><a class="anchor" href="#netlink-sockets"></a><a class="link" href="#netlink-sockets">17.8.7. netlink sockets</a></h4>
<div class="paragraph">
<p>Netlink sockets offer a socket API for kernel / userland communication:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./netlink.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/netlink.c">kernel_modules/netlink.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/netlink.h">lkmc/netlink.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/kernel_modules/netlink.c">userland/kernel_modules/netlink.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/netlink.sh">rootfs_overlay/lkmc/netlink.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Launch multiple user requests in parallel to stress our socket:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod netlink.ko sleep=1
for i in `seq 16`; do ./netlink.out &amp; done</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what is the advantage over <code>read</code>, <code>write</code> and <code>poll</code>? <a href="https://stackoverflow.com/questions/16727212/how-netlink-socket-in-linux-kernel-is-different-from-normal-polling-done-by-appl" class="bare">https://stackoverflow.com/questions/16727212/how-netlink-socket-in-linux-kernel-is-different-from-normal-polling-done-by-appl</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/3299386/how-to-use-netlink-socket-to-communicate-with-a-kernel-module" class="bare">https://stackoverflow.com/questions/3299386/how-to-use-netlink-socket-to-communicate-with-a-kernel-module</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Netlink" class="bare">https://en.wikipedia.org/wiki/Netlink</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kthread"><a class="anchor" href="#kthread"></a><a class="link" href="#kthread">17.9. kthread</a></h3>
<div class="paragraph">
<p>Kernel threads are managed exactly like userland threads; they also have a backing <code>task_struct</code>, and are scheduled with the same mechanism:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod kthread.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/kthread.c">kernel_modules/kthread.c</a></p>
</div>
<div class="paragraph">
<p>Outcome: dmesg counts from <code>0</code> to <code>9</code> once every second infinitely many times:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0
1
2
...
8
9
0
1
2
...</pre>
</div>
</div>
<div class="paragraph">
<p>The count stops when we <code>rmmod</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rmmod kthread</pre>
</div>
</div>
<div class="paragraph">
<p>The sleep is done with <code>usleep_range</code>, see: <a href="#sleep">Section 17.9.2, &#8220;sleep&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/10177641/proper-way-of-handling-threads-in-kernel" class="bare">https://stackoverflow.com/questions/10177641/proper-way-of-handling-threads-in-kernel</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/4084708/how-to-wait-for-a-linux-kernel-thread-kthreadto-exit" class="bare">https://stackoverflow.com/questions/4084708/how-to-wait-for-a-linux-kernel-thread-kthreadto-exit</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="kthreads"><a class="anchor" href="#kthreads"></a><a class="link" href="#kthreads">17.9.1. kthreads</a></h4>
<div class="paragraph">
<p>Let&#8217;s launch two threads and see if they actually run in parallel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod kthreads.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/kthreads.c">kernel_modules/kthreads.c</a></p>
</div>
<div class="paragraph">
<p>Outcome: two threads count to dmesg from <code>0</code> to <code>9</code> in parallel.</p>
</div>
<div class="paragraph">
<p>Each line has output of form:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;thread_id&gt; &lt;count&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>Possible very likely outcome:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>1 0
2 0
1 1
2 1
1 2
2 2
1 3
2 3</pre>
</div>
</div>
<div class="paragraph">
<p>The threads almost always interleaved nicely, thus confirming that they are actually running in parallel.</p>
</div>
</div>
<div class="sect3">
<h4 id="sleep"><a class="anchor" href="#sleep"></a><a class="link" href="#sleep">17.9.2. sleep</a></h4>
<div class="paragraph">
<p>Count to dmesg every one second from <code>0</code> up to <code>n - 1</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod sleep.ko n=5</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/sleep.c">kernel_modules/sleep.c</a></p>
</div>
<div class="paragraph">
<p>The sleep is done with a call to <a href="https://github.com/torvalds/linux/blob/v4.17/kernel/time/timer.c#L1984"><code>usleep_range</code></a> directly inside <code>module_init</code> for simplicity.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/15994603/how-to-sleep-in-the-linux-kernel/44153288#44153288" class="bare">https://stackoverflow.com/questions/15994603/how-to-sleep-in-the-linux-kernel/44153288#44153288</a></p>
</li>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.17/Documentation/timers/timers-howto.txt" class="bare">https://github.com/torvalds/linux/blob/v4.17/Documentation/timers/timers-howto.txt</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="workqueues"><a class="anchor" href="#workqueues"></a><a class="link" href="#workqueues">17.9.3. Workqueues</a></h4>
<div class="paragraph">
<p>A more convenient front-end for <a href="#kthread">kthread</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod workqueue_cheat.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: count from <code>0</code> to <code>9</code> infinitely many times</p>
</div>
<div class="paragraph">
<p>Stop counting:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rmmod workqueue_cheat</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/workqueue_cheat.c">kernel_modules/workqueue_cheat.c</a></p>
</div>
<div class="paragraph">
<p>The workqueue thread is killed after the worker function returns.</p>
</div>
<div class="paragraph">
<p>We can&#8217;t call the module just <code>workqueue.c</code> because there is already a built-in with that name: <a href="https://unix.stackexchange.com/questions/364956/how-can-insmod-fail-with-kernel-module-is-already-loaded-even-is-lsmod-does-not" class="bare">https://unix.stackexchange.com/questions/364956/how-can-insmod-fail-with-kernel-module-is-already-loaded-even-is-lsmod-does-not</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/workqueue.rst" class="bare">https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/workqueue.rst</a></p>
</div>
<div class="sect4">
<h5 id="workqueue-from-workqueue"><a class="anchor" href="#workqueue-from-workqueue"></a><a class="link" href="#workqueue-from-workqueue">17.9.3.1. Workqueue from workqueue</a></h5>
<div class="paragraph">
<p>Count from <code>0</code> to <code>9</code> every second infinitely many times by scheduling a new work item from a work item:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod work_from_work.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Stop:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rmmod work_from_work</pre>
</div>
</div>
<div class="paragraph">
<p>The sleep is done indirectly through: <a href="https://github.com/torvalds/linux/blob/v4.17/include/linux/workqueue.h#L522"><code>queue_delayed_work</code></a>, which waits the specified time before scheduling the work.</p>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/work_from_work.c">kernel_modules/work_from_work.c</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="schedule"><a class="anchor" href="#schedule"></a><a class="link" href="#schedule">17.9.4. schedule</a></h4>
<div class="paragraph">
<p>Let&#8217;s block the entire kernel! Yay:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'dmesg -n 1;insmod schedule.ko schedule=0'</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the system hangs, the only way out is to kill the VM.</p>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/schedule.c">kernel_modules/schedule.c</a></p>
</div>
<div class="paragraph">
<p>kthreads only allow interrupting if you call <code>schedule()</code>, and the <code>schedule=0</code> <a href="#kernel-module-parameters">kernel module parameter</a> turns it off.</p>
</div>
<div class="paragraph">
<p>Sleep functions like <code>usleep_range</code> also end up calling schedule.</p>
</div>
<div class="paragraph">
<p>If we allow <code>schedule()</code> to be called, then the system becomes responsive:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'dmesg -n 1;insmod schedule.ko schedule=1'</pre>
</div>
</div>
<div class="paragraph">
<p>and we can observe the counting with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dmesg -w</pre>
</div>
</div>
<div class="paragraph">
<p>The system also responds if we <a href="#number-of-cores">add another core</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --eval-after 'dmesg -n 1;insmod schedule.ko schedule=0'</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="wait-queues"><a class="anchor" href="#wait-queues"></a><a class="link" href="#wait-queues">17.9.5. Wait queues</a></h4>
<div class="paragraph">
<p>Wait queues are a way to make a thread sleep until an event happens on the queue:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod wait_queue.c</pre>
</div>
</div>
<div class="paragraph">
<p>Dmesg output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0 0
1 0
2 0
# Wait one second.
0 1
1 1
2 1
# Wait one second.
0 2
1 2
2 2
...</pre>
</div>
</div>
<div class="paragraph">
<p>Stop the count:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rmmod wait_queue</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/wait_queue.c">kernel_modules/wait_queue.c</a></p>
</div>
<div class="paragraph">
<p>This example launches three threads:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>one thread generates events every with <a href="https://github.com/torvalds/linux/blob/v4.17/include/linux/wait.h#L195"><code>wake_up</code></a></p>
</li>
<li>
<p>the other two threads wait for that with <a href="https://github.com/torvalds/linux/blob/v4.17/include/linux/wait.h#L286"><code>wait_event</code></a>, and print a dmesg when it happens.</p>
<div class="paragraph">
<p>The <code>wait_event</code> macro works a bit like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>while (!cond)
    sleep_until_event</pre>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="timers"><a class="anchor" href="#timers"></a><a class="link" href="#timers">17.10. Timers</a></h3>
<div class="paragraph">
<p>Count from <code>0</code> to <code>9</code> infinitely many times in 1 second intervals using timers:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod timer.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Stop counting:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rmmod timer</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/timer.c">kernel_modules/timer.c</a></p>
</div>
<div class="paragraph">
<p>Timers are callbacks that run when an interrupt happens, from the interrupt context itself.</p>
</div>
<div class="paragraph">
<p>Therefore they produce more accurate timing than thread scheduling, which is more complex, but you can&#8217;t do too much work inside of them.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/10812858/timers-in-linux-device-drivers" class="bare">https://stackoverflow.com/questions/10812858/timers-in-linux-device-drivers</a></p>
</li>
<li>
<p><a href="https://gist.github.com/yagihiro/310149" class="bare">https://gist.github.com/yagihiro/310149</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="irq"><a class="anchor" href="#irq"></a><a class="link" href="#irq">17.11. IRQ</a></h3>
<div class="sect3">
<h4 id="irq-ko"><a class="anchor" href="#irq-ko"></a><a class="link" href="#irq-ko">17.11.1. irq.ko</a></h4>
<div class="paragraph">
<p>Brute force monitor every shared interrupt that will accept us:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'insmod irq.ko' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/irq.c">kernel_modules/irq.c</a>.</p>
</div>
<div class="paragraph">
<p>Now try the following:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>press a keyboard key and then release it after a few seconds</p>
</li>
<li>
<p>press a mouse key, and release it after a few seconds</p>
</li>
<li>
<p>move the mouse around</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Outcome: dmesg shows which IRQ was fired for each action through messages of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>handler irq = 1 dev = 250</pre>
</div>
</div>
<div class="paragraph">
<p><code>dev</code> is the character device for the module and never changes, as can be confirmed by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep lkmc_irq /proc/devices</pre>
</div>
</div>
<div class="paragraph">
<p>The IRQs that we observe are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>1</code> for keyboard press and release.</p>
<div class="paragraph">
<p>If you hold the key down for a while, it starts firing at a constant rate. So this happens at the hardware level!</p>
</div>
</li>
<li>
<p><code>12</code> mouse actions</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This only works if for IRQs for which the other handlers are registered as <code>IRQF_SHARED</code>.</p>
</div>
<div class="paragraph">
<p>We can see which ones are those, either via dmesg messages of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>genirq: Flags mismatch irq 0. 00000080 (myirqhandler0) vs. 00015a00 (timer)
request_irq irq = 0 ret = -16
request_irq irq = 1 ret = 0</pre>
</div>
</div>
<div class="paragraph">
<p>which indicate that <code>0</code> is not, but <code>1</code> is, or with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/interrupts</pre>
</div>
</div>
<div class="paragraph">
<p>which shows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  0:         31   IO-APIC   2-edge      timer
  1:          9   IO-APIC   1-edge      i8042, myirqhandler0</pre>
</div>
</div>
<div class="paragraph">
<p>so only <code>1</code> has <code>myirqhandler0</code> attached but not <code>0</code>.</p>
</div>
<div class="paragraph">
<p>The <a href="#qemu-monitor">QEMU monitor</a> also has some interrupt statistics for x86_64:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor info irq</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: properly understand how each IRQ maps to what number.</p>
</div>
</div>
<div class="sect3">
<h4 id="dummy-irq"><a class="anchor" href="#dummy-irq"></a><a class="link" href="#dummy-irq">17.11.2. dummy-irq</a></h4>
<div class="paragraph">
<p>The Linux kernel v4.16 mainline also has a <code>dummy-irq</code> module at <code>drivers/misc/dummy-irq.c</code> for monitoring a single IRQ.</p>
</div>
<div class="paragraph">
<p>We build it by default with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CONFIG_DUMMY_IRQ=m</pre>
</div>
</div>
<div class="paragraph">
<p>And then you can do</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>and in guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modprobe dummy-irq irq=1</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: when you click a key on the keyboard, dmesg shows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dummy-irq: interrupt occurred on IRQ 1</pre>
</div>
</div>
<div class="paragraph">
<p>However, this module is intended to fire only once as can be seen from its source:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    static int count = 0;

    if (count == 0) {
        printk(KERN_INFO "dummy-irq: interrupt occurred on IRQ %d\n",
            irq);
        count++;
    }</pre>
</div>
</div>
<div class="paragraph">
<p>and furthermore interrupt <code>1</code> and <code>12</code> happen immediately TODO why, were they somehow pending?</p>
</div>
</div>
<div class="sect3">
<h4 id="procinterrupts"><a class="anchor" href="#procinterrupts"></a><a class="link" href="#procinterrupts">17.11.3. /proc/interrupts</a></h4>
<div class="paragraph">
<p>In the guest with <a href="#qemu-graphic-mode">QEMU graphic mode</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>watch -n 1 cat /proc/interrupts</pre>
</div>
</div>
<div class="paragraph">
<p>Then see how clicking the mouse and keyboard affect the interrupt counts.</p>
</div>
<div class="paragraph">
<p>This confirms that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>1: keyboard</p>
</li>
<li>
<p>12: mouse click and drags</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The module also shows which handlers are registered for each IRQ, as we have observed at <a href="#irq-ko">irq.ko</a></p>
</div>
<div class="paragraph">
<p>When in text mode, we can also observe interrupt line 4 with handler <code>ttyS0</code> increase continuously as IO goes through the UART.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="kernel-utility-functions"><a class="anchor" href="#kernel-utility-functions"></a><a class="link" href="#kernel-utility-functions">17.12. Kernel utility functions</a></h3>
<div class="paragraph">
<p><a href="https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/kernel-api.rst" class="bare">https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/kernel-api.rst</a></p>
</div>
<div class="sect3">
<h4 id="kstrto"><a class="anchor" href="#kstrto"></a><a class="link" href="#kstrto">17.12.1. kstrto</a></h4>
<div class="paragraph">
<p>Convert a string to an integer:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./kstrto.sh
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/kstrto.c">kernel_modules/kstrto.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/kstrto.sh">rootfs_overlay/lkmc/kstrto.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/6139493/how-convert-char-to-int-in-linux-kernel/49811658#49811658" class="bare">https://stackoverflow.com/questions/6139493/how-convert-char-to-int-in-linux-kernel/49811658#49811658</a></p>
</div>
</div>
<div class="sect3">
<h4 id="virt-to-phys"><a class="anchor" href="#virt-to-phys"></a><a class="link" href="#virt-to-phys">17.12.2. virt_to_phys</a></h4>
<div class="paragraph">
<p>Convert a virtual address to physical:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod virt_to_phys.ko
cat /sys/kernel/debug/lkmc_virt_to_phys</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/virt_to_phys.c">kernel_modules/virt_to_phys.c</a></p>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>*kmalloc_ptr = 0x12345678
kmalloc_ptr = ffff88000e169ae8
virt_to_phys(kmalloc_ptr) = 0xe169ae8
static_var = 0x12345678
&amp;static_var = ffffffffc0002308
virt_to_phys(&amp;static_var) = 0x40002308</pre>
</div>
</div>
<div class="paragraph">
<p>We can confirm that the <code>kmalloc_ptr</code> translation worked with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor 'xp 0xe169ae8'</pre>
</div>
</div>
<div class="paragraph">
<p>which reads four bytes from a given physical address, and gives the expected:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>000000000e169ae8: 0x12345678</pre>
</div>
</div>
<div class="paragraph">
<p>TODO it only works for kmalloc however, for the static variable:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor 'xp 0x40002308'</pre>
</div>
</div>
<div class="paragraph">
<p>it gave a wrong value of <code>00000000</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/5748492/is-there-any-api-for-determining-the-physical-address-from-virtual-address-in-li/45128487#45128487" class="bare">https://stackoverflow.com/questions/5748492/is-there-any-api-for-determining-the-physical-address-from-virtual-address-in-li/45128487#45128487</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/39134990/mmap-of-dev-mem-fails-with-invalid-argument-for-virt-to-phys-address-but-addre/45127582#45127582" class="bare">https://stackoverflow.com/questions/39134990/mmap-of-dev-mem-fails-with-invalid-argument-for-virt-to-phys-address-but-addre/45127582#45127582</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/43325205/can-we-use-virt-to-phys-for-user-space-memory-in-kernel-module" class="bare">https://stackoverflow.com/questions/43325205/can-we-use-virt-to-phys-for-user-space-memory-in-kernel-module</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="userland-physical-address-experiments"><a class="anchor" href="#userland-physical-address-experiments"></a><a class="link" href="#userland-physical-address-experiments">17.12.2.1. Userland physical address experiments</a></h5>
<div class="paragraph">
<p>Only tested in x86_64.</p>
</div>
<div class="paragraph">
<p>The Linux kernel exposes physical addresses to userland through:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/proc/&lt;pid&gt;/maps</code></p>
</li>
<li>
<p><code>/proc/&lt;pid&gt;/pagemap</code></p>
</li>
<li>
<p><code>/dev/mem</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In this section we will play with them.</p>
</div>
<div class="paragraph">
<p>The following files contain examples to access that data and test it out:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/pagemap.h">lkmc/pagemap.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/virt_to_phys.sh">rootfs_overlay/lkmc/virt_to_phys.sh</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/virt_to_phys_user.c">userland/linux/virt_to_phys_user.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/virt_to_phys_test.c">userland/posix/virt_to_phys_test.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>First get a virtual address to play with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./posix/virt_to_phys_test.out &amp;</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/virt_to_phys_test.c">userland/posix/virt_to_phys_test.c</a></p>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vaddr 0x600800
pid 110</pre>
</div>
</div>
<div class="paragraph">
<p>The program:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>allocates a <code>volatile</code> variable and sets is value to <code>0x12345678</code></p>
</li>
<li>
<p>prints the virtual address of the variable, and the program PID</p>
</li>
<li>
<p>runs a while loop until until the value of the variable gets mysteriously changed somehow, e.g. by nasty tinkerers like us</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, translate the virtual address to physical using <code>/proc/&lt;pid&gt;/maps</code> and <code>/proc/&lt;pid&gt;/pagemap</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./linux/virt_to_phys_user.out 110 0x600800</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output physical address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x7c7b800</pre>
</div>
</div>
<div class="paragraph">
<p>Now we can verify that <code>linux/virt_to_phys_user.out</code> gave the correct physical address in the following ways:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#qemu-xp">QEMU xp</a></p>
</li>
<li>
<p><a href="#dev-mem">/dev/mem</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/17021214/decode-proc-pid-pagemap-entry/45126141#45126141" class="bare">https://stackoverflow.com/questions/17021214/decode-proc-pid-pagemap-entry/45126141#45126141</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/6284810/proc-pid-pagemaps-and-proc-pid-maps-linux/45500208#45500208" class="bare">https://stackoverflow.com/questions/6284810/proc-pid-pagemaps-and-proc-pid-maps-linux/45500208#45500208</a></p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="qemu-xp"><a class="anchor" href="#qemu-xp"></a><a class="link" href="#qemu-xp">17.12.2.1.1. QEMU xp</a></h6>
<div class="paragraph">
<p>The <code>xp</code> <a href="#qemu-monitor">QEMU monitor</a> command reads memory at a given physical address.</p>
</div>
<div class="paragraph">
<p>First launch <code>linux/virt_to_phys_user.out</code> as described at <a href="#userland-physical-address-experiments">Userland physical address experiments</a>.</p>
</div>
<div class="paragraph">
<p>On a second terminal, use QEMU to read the physical address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor 'xp 0x7c7b800'</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0000000007c7b800: 0x12345678</pre>
</div>
</div>
<div class="paragraph">
<p>Yes!!! We read the correct value from the physical address.</p>
</div>
<div class="paragraph">
<p>We could not find however to write to memory from the QEMU monitor, boring.</p>
</div>
</div>
<div class="sect5">
<h6 id="dev-mem"><a class="anchor" href="#dev-mem"></a><a class="link" href="#dev-mem">17.12.2.1.2. /dev/mem</a></h6>
<div class="paragraph">
<p><code>/dev/mem</code> exposes access to physical addresses, and we use it through the convenient <code>devmem</code> BusyBox utility.</p>
</div>
<div class="paragraph">
<p>First launch <code>linux/virt_to_phys_user.out</code> as described at <a href="#userland-physical-address-experiments">Userland physical address experiments</a>.</p>
</div>
<div class="paragraph">
<p>Next, read from the physical address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>devmem 0x7c7b800</pre>
</div>
</div>
<div class="paragraph">
<p>Possible output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Memory mapped at address 0x7ff7dbe01000.
Value at address 0X7C7B800 (0x7ff7dbe01800): 0x12345678</pre>
</div>
</div>
<div class="paragraph">
<p>which shows that the physical memory contains the expected value <code>0x12345678</code>.</p>
</div>
<div class="paragraph">
<p><code>0x7ff7dbe01000</code> is a new virtual address that <code>devmem</code> maps to the physical address to be able to read from it.</p>
</div>
<div class="paragraph">
<p>Modify the physical memory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>devmem 0x7c7b800 w 0x9abcdef0</pre>
</div>
</div>
<div class="paragraph">
<p>After one second, we see on the screen:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i 9abcdef0
[1]+  Done                       ./posix/virt_to_phys_test.out</pre>
</div>
</div>
<div class="paragraph">
<p>so the value changed, and the <code>while</code> loop exited!</p>
</div>
<div class="paragraph">
<p>This example requires:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>CONFIG_STRICT_DEVMEM=n</code>, otherwise <code>devmem</code> fails with:</p>
<div class="literalblock">
<div class="content">
<pre>devmem: mmap: Operation not permitted</pre>
</div>
</div>
</li>
<li>
<p><code>nopat</code> kernel parameter</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>which we set by default.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/11891979/how-to-access-mmaped-dev-mem-without-crashing-the-linux-kernel" class="bare">https://stackoverflow.com/questions/11891979/how-to-access-mmaped-dev-mem-without-crashing-the-linux-kernel</a></p>
</div>
</div>
<div class="sect5">
<h6 id="pagemap-dump-out"><a class="anchor" href="#pagemap-dump-out"></a><a class="link" href="#pagemap-dump-out">17.12.2.1.3. pagemap_dump.out</a></h6>
<div class="paragraph">
<p>Dump the physical address of all pages mapped to a given process using <code>/proc/&lt;pid&gt;/maps</code> and <code>/proc/&lt;pid&gt;/pagemap</code>.</p>
</div>
<div class="paragraph">
<p>First launch <code>linux/virt_to_phys_user.out</code> as described at <a href="#userland-physical-address-experiments">Userland physical address experiments</a>. Suppose that the output was:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># ./posix/virt_to_phys_test.out &amp;
vaddr 0x601048
pid 63
# ./linux/virt_to_phys_user.out 63 0x601048
0x1a61048</pre>
</div>
</div>
<div class="paragraph">
<p>Now obtain the page map for the process:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./linux/pagemap_dump.out 63</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output excerpt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vaddr pfn soft-dirty file/shared swapped present library
400000 1ede 0 1 0 1 ./posix/virt_to_phys_test.out
600000 1a6f 0 0 0 1 ./posix/virt_to_phys_test.out
601000 1a61 0 0 0 1 ./posix/virt_to_phys_test.out
602000 2208 0 0 0 1 [heap]
603000 220b 0 0 0 1 [heap]
7ffff78ec000 1fd4 0 1 0 1 /lib/libuClibc-1.0.30.so</pre>
</div>
</div>
<div class="paragraph">
<p>Source:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/pagemap_dump.c">userland/linux/pagemap_dump.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/pagemap.h">lkmc/pagemap.h</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Adapted from: <a href="https://github.com/dwks/pagemap/blob/8a25747bc79d6080c8b94eac80807a4dceeda57a/pagemap2.c" class="bare">https://github.com/dwks/pagemap/blob/8a25747bc79d6080c8b94eac80807a4dceeda57a/pagemap2.c</a></p>
</div>
<div class="paragraph">
<p>Meaning of the flags:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>vaddr</code>: first virtual address of a page the belongs to the process. Notably:</p>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- -l "$(./getvar userland_build_dir)/posix/virt_to_phys_test.out"</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
...
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000075c 0x000000000000075c  R E    0x200000
  LOAD           0x0000000000000e98 0x0000000000600e98 0x0000000000600e98
                 0x00000000000001b4 0x0000000000000218  RW     0x200000

 Section to Segment mapping:
  Segment Sections...
...
   02     .interp .hash .dynsym .dynstr .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame
   03     .ctors .dtors .jcr .dynamic .got.plt .data .bss</pre>
</div>
</div>
<div class="paragraph">
<p>from which we deduce that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>400000</code> is the text segment</p>
</li>
<li>
<p><code>600000</code> is the data segment</p>
</li>
</ul>
</div>
</li>
<li>
<p><code>pfn</code>: add three zeroes to it, and you have the physical address.</p>
<div class="paragraph">
<p>Three zeroes is 12 bits which is 4kB, which is the size of a page.</p>
</div>
<div class="paragraph">
<p>For example, the virtual address <code>0x601000</code> has <code>pfn</code> of <code>0x1a61</code>, which means that its physical address is <code>0x1a61000</code></p>
</div>
<div class="paragraph">
<p>This is consistent with what <code>linux/virt_to_phys_user.out</code> told us: the virtual address <code>0x601048</code> has physical address <code>0x1a61048</code>.</p>
</div>
<div class="paragraph">
<p><code>048</code> corresponds to the three last zeroes, and is the offset within the page.</p>
</div>
<div class="paragraph">
<p>Also, this value falls inside <code>0x601000</code>, which as previously analyzed is the data section, which is the normal location for global variables such as ours.</p>
</div>
</li>
<li>
<p><code>soft-dirty</code>: TODO</p>
</li>
<li>
<p><code>file/shared</code>: TODO. <code>1</code> seems to indicate that the page can be shared across processes, possibly for read-only pages? E.g. the text segment has <code>1</code>, but the data has <code>0</code>.</p>
</li>
<li>
<p><code>swapped</code>: TODO swapped to disk?</p>
</li>
<li>
<p><code>present</code>: TODO vs swapped?</p>
</li>
<li>
<p><code>library</code>: which executable owns that page</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This program works in two steps:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>parse the human readable lines lines from <code>/proc/&lt;pid&gt;/maps</code>. This files contains lines of form:</p>
<div class="literalblock">
<div class="content">
<pre>7ffff7b6d000-7ffff7bdd000 r-xp 00000000 fe:00 658                        /lib/libuClibc-1.0.22.so</pre>
</div>
</div>
<div class="paragraph">
<p>which tells us that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>7f8af99f8000-7f8af99ff000</code> is a virtual address range that belong to the process, possibly containing multiple pages.</p>
</li>
<li>
<p><code>/lib/libuClibc-1.0.22.so</code> is the name of the library that owns that memory</p>
</li>
</ul>
</div>
</li>
<li>
<p>loop over each page of each address range, and ask <code>/proc/&lt;pid&gt;/pagemap</code> for more information about that page, including the physical address</p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-tracing"><a class="anchor" href="#linux-kernel-tracing"></a><a class="link" href="#linux-kernel-tracing">17.13. Linux kernel tracing</a></h3>
<div class="paragraph">
<p>Good overviews:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html" class="bare">http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html</a> by Brendan Greg, AKA the master of tracing. Also: <a href="https://github.com/brendangregg/perf-tools" class="bare">https://github.com/brendangregg/perf-tools</a></p>
</li>
<li>
<p><a href="https://jvns.ca/blog/2017/07/05/linux-tracing-systems/" class="bare">https://jvns.ca/blog/2017/07/05/linux-tracing-systems/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>I hope to have examples of all methods some day, since I&#8217;m obsessed with visibility.</p>
</div>
<div class="sect3">
<h4 id="config-proc-events"><a class="anchor" href="#config-proc-events"></a><a class="link" href="#config-proc-events">17.13.1. CONFIG_PROC_EVENTS</a></h4>
<div class="paragraph">
<p>Logs proc events such as process creation to a <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/netlink.c">netlink socket</a>.</p>
</div>
<div class="paragraph">
<p>We then have a userland program that listens to the events and prints them out:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># ./linux/proc_events.out &amp;
# set mcast listen ok
# sleep 2 &amp; sleep 1
fork: parent tid=48 pid=48 -&gt; child tid=79 pid=79
fork: parent tid=48 pid=48 -&gt; child tid=80 pid=80
exec: tid=80 pid=80
exec: tid=79 pid=79
# exit: tid=80 pid=80 exit_code=0
exit: tid=79 pid=79 exit_code=0
echo a
a
#</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/proc_events.c">userland/linux/proc_events.c</a></p>
</div>
<div class="paragraph">
<p>TODO: why <code>exit: tid=79</code> shows after <code>exit: tid=80</code>?</p>
</div>
<div class="paragraph">
<p>Note how <code>echo a</code> is a Bash built-in, and therefore does not spawn a new process.</p>
</div>
<div class="paragraph">
<p>TODO: why does this produce no output?</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./linux/proc_events.out &gt;f &amp;</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/6075013/detect-launching-of-programs-on-linux-platform/8255487#8255487" class="bare">https://stackoverflow.com/questions/6075013/detect-launching-of-programs-on-linux-platform/8255487#8255487</a></p>
</li>
<li>
<p><a href="https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn" class="bare">https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/260162/how-to-track-newly-created-processes" class="bare">https://unix.stackexchange.com/questions/260162/how-to-track-newly-created-processes</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO can you get process data such as UID and process arguments? It seems not since <code>exec_proc_event</code> contains so little data: <a href="https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80" class="bare">https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80</a> We could try to immediately read it from <code>/proc</code>, but there is a risk that the process finished and another one took its PID, so it wouldn&#8217;t be reliable.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/163681/print-pids-and-names-of-processes-as-they-are-created/163689" class="bare">https://unix.stackexchange.com/questions/163681/print-pids-and-names-of-processes-as-they-are-created/163689</a> requests process name</p>
</li>
<li>
<p><a href="https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn" class="bare">https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn</a> requests UID</p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="config-proc-events-aarch64"><a class="anchor" href="#config-proc-events-aarch64"></a><a class="link" href="#config-proc-events-aarch64">17.13.1.1. CONFIG_PROC_EVENTS aarch64</a></h5>
<div class="paragraph">
<p>0111ca406bdfa6fd65a2605d353583b4c4051781 was failing with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&gt;&gt;&gt; kernel_modules 1.0 Building
/usr/bin/make -j8 -C '/linux-kernel-module-cheat//out/aarch64/buildroot/build/kernel_modules-1.0/user' BR2_PACKAGE_OPENBLAS="" CC="/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-gcc" LD="/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-ld"
/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-gcc  -ggdb3 -fopenmp -O0 -std=c99 -Wall -Werror -Wextra -o 'proc_events.out' 'proc_events.c'
In file included from /linux-kernel-module-cheat//out/aarch64/buildroot/host/aarch64-buildroot-linux-uclibc/sysroot/usr/include/signal.h:329:0,
                 from proc_events.c:12:
/linux-kernel-module-cheat//out/aarch64/buildroot/host/aarch64-buildroot-linux-uclibc/sysroot/usr/include/sys/ucontext.h:50:16: error: field ‘uc_mcontext’ has incomplete type
     mcontext_t uc_mcontext;
                ^~~~~~~~~~~</pre>
</div>
</div>
<div class="paragraph">
<p>so we commented it out.</p>
</div>
<div class="paragraph">
<p>Related threads:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://mailman.uclibc-ng.org/pipermail/devel/2018-January/001624.html" class="bare">https://mailman.uclibc-ng.org/pipermail/devel/2018-January/001624.html</a></p>
</li>
<li>
<p><a href="https://github.com/DynamoRIO/dynamorio/issues/2356" class="bare">https://github.com/DynamoRIO/dynamorio/issues/2356</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If we try to naively update uclibc to 1.0.29 with <code>buildroot_override</code>, which contains the above mentioned patch, clean <code>aarch64</code> test build fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>../utils/ldd.c: In function 'elf_find_dynamic':
../utils/ldd.c:238:12: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     return (void *)byteswap_to_host(dynp-&gt;d_un.d_val);
            ^
/tmp/user/20321/cciGScKB.o: In function `process_line_callback':
msgmerge.c:(.text+0x22): undefined reference to `escape'
/tmp/user/20321/cciGScKB.o: In function `process':
msgmerge.c:(.text+0xf6): undefined reference to `poparser_init'
msgmerge.c:(.text+0x11e): undefined reference to `poparser_feed_line'
msgmerge.c:(.text+0x128): undefined reference to `poparser_finish'
collect2: error: ld returned 1 exit status
Makefile.in:120: recipe for target '../utils/msgmerge.host' failed
make[2]: *** [../utils/msgmerge.host] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/user/20321/ccF8V8jF.o: In function `process':
msgfmt.c:(.text+0xbf3): undefined reference to `poparser_init'
msgfmt.c:(.text+0xc1f): undefined reference to `poparser_feed_line'
msgfmt.c:(.text+0xc2b): undefined reference to `poparser_finish'
collect2: error: ld returned 1 exit status
Makefile.in:120: recipe for target '../utils/msgfmt.host' failed
make[2]: *** [../utils/msgfmt.host] Error 1
package/pkg-generic.mk:227: recipe for target '/data/git/linux-kernel-module-cheat/out/aarch64/buildroot/build/uclibc-custom/.stamp_built' failed
make[1]: *** [/data/git/linux-kernel-module-cheat/out/aarch64/buildroot/build/uclibc-custom/.stamp_built] Error 2
Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2</pre>
</div>
</div>
<div class="paragraph">
<p>Buildroot master has already moved to uclibc 1.0.29 at f8546e836784c17aa26970f6345db9d515411700, but it is not yet in any tag&#8230;&#8203; so I&#8217;m not tempted to update it yet just for this.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="ftrace"><a class="anchor" href="#ftrace"></a><a class="link" href="#ftrace">17.13.2. ftrace</a></h4>
<div class="paragraph">
<p>Trace a single function:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd /sys/kernel/debug/tracing/

# Stop tracing.
echo 0 &gt; tracing_on

# Clear previous trace.
echo &gt; trace

# List the available tracers, and pick one.
cat available_tracers
echo function &gt; current_tracer

# List all functions that can be traced
# cat available_filter_functions
# Choose one.
echo __kmalloc &gt; set_ftrace_filter
# Confirm that only __kmalloc is enabled.
cat enabled_functions

echo 1 &gt; tracing_on

# Latest events.
head trace

# Observe trace continuously, and drain seen events out.
cat trace_pipe &amp;</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># tracer: function
#
# entries-in-buffer/entries-written: 97/97   #P:1
#
#                              _-----=&gt; irqs-off
#                             / _----=&gt; need-resched
#                            | / _---=&gt; hardirq/softirq
#                            || / _--=&gt; preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
            head-228   [000] ....   825.534637: __kmalloc &lt;-load_elf_phdrs
            head-228   [000] ....   825.534692: __kmalloc &lt;-load_elf_binary
            head-228   [000] ....   825.534815: __kmalloc &lt;-load_elf_phdrs
            head-228   [000] ....   825.550917: __kmalloc &lt;-__seq_open_private
            head-228   [000] ....   825.550953: __kmalloc &lt;-tracing_open
            head-229   [000] ....   826.756585: __kmalloc &lt;-load_elf_phdrs
            head-229   [000] ....   826.756627: __kmalloc &lt;-load_elf_binary
            head-229   [000] ....   826.756719: __kmalloc &lt;-load_elf_phdrs
            head-229   [000] ....   826.773796: __kmalloc &lt;-__seq_open_private
            head-229   [000] ....   826.773835: __kmalloc &lt;-tracing_open
            head-230   [000] ....   827.174988: __kmalloc &lt;-load_elf_phdrs
            head-230   [000] ....   827.175046: __kmalloc &lt;-load_elf_binary
            head-230   [000] ....   827.175171: __kmalloc &lt;-load_elf_phdrs</pre>
</div>
</div>
<div class="paragraph">
<p>Trace all possible functions, and draw a call graph:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; max_graph_depth
echo 1 &gt; events/enable
echo function_graph &gt; current_tracer</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
 0)   2.173 us    |                  } /* ntp_tick_length */
 0)               |                  timekeeping_update() {
 0)   4.176 us    |                    ntp_get_next_leap();
 0)   5.016 us    |                    update_vsyscall();
 0)               |                    raw_notifier_call_chain() {
 0)   2.241 us    |                      notifier_call_chain();
 0) + 19.879 us   |                    }
 0)   3.144 us    |                    update_fast_timekeeper();
 0)   2.738 us    |                    update_fast_timekeeper();
 0) ! 117.147 us  |                  }
 0)               |                  _raw_spin_unlock_irqrestore() {
 0)   4.045 us    |                    _raw_write_unlock_irqrestore();
 0) + 22.066 us   |                  }
 0) ! 265.278 us  |                } /* update_wall_time */</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what do <code>+</code> and <code>!</code> mean?</p>
</div>
<div class="paragraph">
<p>Each <code>enable</code> under the <code>events/</code> tree enables a certain set of functions, the higher the <code>enable</code> more functions are enabled.</p>
</div>
<div class="paragraph">
<p>TODO: can you get function arguments? <a href="https://stackoverflow.com/questions/27608752/does-ftrace-allow-capture-of-system-call-arguments-to-the-linux-kernel-or-only" class="bare">https://stackoverflow.com/questions/27608752/does-ftrace-allow-capture-of-system-call-arguments-to-the-linux-kernel-or-only</a></p>
</div>
<div class="sect4">
<h5 id="ftrace-system-calls"><a class="anchor" href="#ftrace-system-calls"></a><a class="link" href="#ftrace-system-calls">17.13.2.1. ftrace system calls</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/29840213/how-do-i-trace-a-system-call-in-linux/51856306#51856306" class="bare">https://stackoverflow.com/questions/29840213/how-do-i-trace-a-system-call-in-linux/51856306#51856306</a></p>
</div>
</div>
<div class="sect4">
<h5 id="trace-cmd"><a class="anchor" href="#trace-cmd"></a><a class="link" href="#trace-cmd">17.13.2.2. trace-cmd</a></h5>
<div class="paragraph">
<p>TODO example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_TRACE_CMD=y'</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="kprobes"><a class="anchor" href="#kprobes"></a><a class="link" href="#kprobes">17.13.3. Kprobes</a></h4>
<div class="paragraph">
<p>kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-)</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --config 'CONFIG_KPROBES=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Then on guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod kprobe_example.ko
sleep 4 &amp; sleep 4 &amp;'</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: dmesg outputs on every fork:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;_do_fork&gt; pre_handler: p-&gt;addr = 0x00000000e1360063, ip = ffffffff810531d1, flags = 0x246
&lt;_do_fork&gt; post_handler: p-&gt;addr = 0x00000000e1360063, flags = 0x246
&lt;_do_fork&gt; pre_handler: p-&gt;addr = 0x00000000e1360063, ip = ffffffff810531d1, flags = 0x246
&lt;_do_fork&gt; post_handler: p-&gt;addr = 0x00000000e1360063, flags = 0x246</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/kprobe_example.c">kernel_modules/kprobe_example.c</a></p>
</div>
<div class="paragraph">
<p>TODO: it does not work if I try to immediately launch <code>sleep</code>, why?</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod kprobe_example.ko
sleep 4 &amp; sleep 4 &amp;</pre>
</div>
</div>
<div class="paragraph">
<p>I don&#8217;t think your code can refer to the surrounding kernel code however: the only visible thing is the value of the registers.</p>
</div>
<div class="paragraph">
<p>You can then hack it up to read the stack and read argument values, but do you really want to?</p>
</div>
<div class="paragraph">
<p>There is also a kprobes + ftrace based mechanism with <code>CONFIG_KPROBE_EVENTS=y</code> which does read the memory for us based on format strings that indicate type&#8230;&#8203; <a href="https://github.com/torvalds/linux/blob/v4.16/Documentation/trace/kprobetrace.txt" class="bare">https://github.com/torvalds/linux/blob/v4.16/Documentation/trace/kprobetrace.txt</a> Horrendous. Used by: <a href="https://github.com/brendangregg/perf-tools/blob/98d42a2a1493d2d1c651a5c396e015d4f082eb20/execsnoop" class="bare">https://github.com/brendangregg/perf-tools/blob/98d42a2a1493d2d1c651a5c396e015d4f082eb20/execsnoop</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.16/Documentation/kprobes.txt" class="bare">https://github.com/torvalds/linux/blob/v4.16/Documentation/kprobes.txt</a></p>
</li>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.17/samples/kprobes/kprobe_example.c" class="bare">https://github.com/torvalds/linux/blob/v4.17/samples/kprobes/kprobe_example.c</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="count-boot-instructions"><a class="anchor" href="#count-boot-instructions"></a><a class="link" href="#count-boot-instructions">17.13.4. Count boot instructions</a></h4>
<div class="paragraph">
<p>TODO: didn&#8217;t port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.quora.com/How-many-instructions-does-a-typical-Linux-kernel-boot-take" class="bare">https://www.quora.com/How-many-instructions-does-a-typical-Linux-kernel-boot-take</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/chat/issues/31" class="bare">https://github.com/cirosantilli/chat/issues/31</a></p>
</li>
<li>
<p><a href="https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/" class="bare">https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/</a></p>
</li>
<li>
<p><code>qemu/docs/tracing.txt</code> and <code>qemu/docs/replay.txt</code></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/39149446/how-to-use-qemus-simple-trace-backend/46497873#46497873" class="bare">https://stackoverflow.com/questions/39149446/how-to-use-qemus-simple-trace-backend/46497873#46497873</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Results (boot not excluded) are shown at: <a href="#table-boot-instruction-counts">Table 1, &#8220;Boot instruction counts for various setups&#8221;</a></p>
</div>
<table id="table-boot-instruction-counts" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 1. Boot instruction counts for various setups</caption>
<colgroup>
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Commit</th>
<th class="tableblock halign-left valign-top">Arch</th>
<th class="tableblock halign-left valign-top">Simulator</th>
<th class="tableblock halign-left valign-top">Instruction count</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7228f75ac74c896417fb8c5ba3d375a14ed4d36b</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">arm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">QEMU</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">680k</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7228f75ac74c896417fb8c5ba3d375a14ed4d36b</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">arm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 AtomicSimpleCPU</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">160M</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7228f75ac74c896417fb8c5ba3d375a14ed4d36b</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">arm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 HPI</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">155M</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7228f75ac74c896417fb8c5ba3d375a14ed4d36b</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">x86_64</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">QEMU</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">3M</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">7228f75ac74c896417fb8c5ba3d375a14ed4d36b</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">x86_64</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 AtomicSimpleCPU</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">528M</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>QEMU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./trace-boot --arch x86_64</pre>
</div>
</div>
<div class="paragraph">
<p>sample output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>instructions 1833863
entry_address 0x1000000
instructions_firmware 20708</pre>
</div>
</div>
<div class="paragraph">
<p>gem5:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --eval 'm5 exit'
# Or:
# ./run --arch aarch64 --emulator gem5 --eval 'm5 exit' -- --cpu-type=HPI --caches
./gem5-stat --arch aarch64 sim_insts</pre>
</div>
</div>
<div class="paragraph">
<p>Notes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>0x1000000</code> is the address where QEMU puts the Linux kernel at with <code>-kernel</code> in x86.</p>
<div class="paragraph">
<p>It can be found from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain readelf -- -e "$(./getvar vmlinux)" | grep Entry</pre>
</div>
</div>
<div class="paragraph">
<p>TODO confirm further. If I try to break there with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb *0x1000000</pre>
</div>
</div>
<div class="paragraph">
<p>but I have no corresponding source line. Also note that this line is not actually the first line, since the kernel messages such as <code>early console in extract_kernel</code> have already shown on screen at that point. This does not break at all:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb extract_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>It only appears once on every log I&#8217;ve seen so far, checked with <code>grep 0x1000000 trace.txt</code></p>
</div>
<div class="paragraph">
<p>Then when we count the instructions that run before the kernel entry point, there is only about 100k instructions, which is insignificant compared to the kernel boot itself.</p>
</div>
<div class="paragraph">
<p>TODO <code>--arch arm</code> and <code>--arch aarch64</code> does not count firmware instructions properly because the entry point address of the ELF file (<code>ffffff8008080000</code> for <code>aarch64</code>) does not show up on the trace at all. Tested on <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/f8c0502bb2680f2dbe7c1f3d7958f60265347005">f8c0502bb2680f2dbe7c1f3d7958f60265347005</a>.</p>
</div>
</li>
<li>
<p>We can also discount the instructions after <code>init</code> runs by using <code>readelf</code> to get the initial address of <code>init</code>. One easy way to do that now is to just run:</p>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --userland "$(./getvar userland_build_dir)/linux/poweroff.out" main</pre>
</div>
</div>
<div class="paragraph">
<p>And get that from the traces, e.g. if the address is <code>4003a0</code>, then we search:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep -n 4003a0 trace.txt</pre>
</div>
</div>
<div class="paragraph">
<p>I have observed a single match for that instruction, so it must be the init, and there were only 20k instructions after it, so the impact is negligible.</p>
</div>
</li>
<li>
<p>to disable networking. Is replacing <code>init</code> enough?</p>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/181254/how-do-you-boot-linux-with-networking-disabled" class="bare">https://superuser.com/questions/181254/how-do-you-boot-linux-with-networking-disabled</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015" class="bare">https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p><code>CONFIG_NET=n</code> did not significantly reduce instruction counts, so maybe replacing <code>init</code> is enough.</p>
</div>
</li>
<li>
<p>gem5 simulates memory latencies. So I think that the CPU loops idle while waiting for memory, and counts will be higher.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-hardening"><a class="anchor" href="#linux-kernel-hardening"></a><a class="link" href="#linux-kernel-hardening">17.14. Linux kernel hardening</a></h3>
<div class="paragraph">
<p>Make it harder to get hacked and easier to notice that you were, at the cost of some (small?) runtime overhead.</p>
</div>
<div class="sect3">
<h4 id="config-fortify-source"><a class="anchor" href="#config-fortify-source"></a><a class="link" href="#config-fortify-source">17.14.1. CONFIG_FORTIFY_SOURCE</a></h4>
<div class="paragraph">
<p>Detects buffer overflows for us:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux --config 'CONFIG_FORTIFY_SOURCE=y' --linux-build-id fortify
./build-modules --clean
./build-modules
./build-buildroot
./run --eval-after 'insmod strlen_overflow.ko' --linux-build-id fortify</pre>
</div>
</div>
<div class="paragraph">
<p>Possible dmesg output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>strlen_overflow: loading out-of-tree module taints kernel.
detected buffer overflow in strlen
------------[ cut here ]------------</pre>
</div>
</div>
<div class="paragraph">
<p>followed by a trace.</p>
</div>
<div class="paragraph">
<p>You may not get this error because this depends on <code>strlen</code> overflowing at least until the next page: if a random <code>\0</code> appears soon enough, it won&#8217;t blow up as desired.</p>
</div>
<div class="paragraph">
<p>TODO not always reproducible. Find a more reproducible failure. I could not observe it on:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod memcpy_overflow.ko</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/strlen_overflow.c">kernel_modules/strlen_overflow.c</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://www.reddit.com/r/hacking/comments/8h4qxk/what_a_buffer_overflow_in_the_linux_kernel_looks/" class="bare">https://www.reddit.com/r/hacking/comments/8h4qxk/what_a_buffer_overflow_in_the_linux_kernel_looks/</a></p>
</div>
</div>
<div class="sect3">
<h4 id="linux-security-modules"><a class="anchor" href="#linux-security-modules"></a><a class="link" href="#linux-security-modules">17.14.2. Linux security modules</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Linux_Security_Modules" class="bare">https://en.wikipedia.org/wiki/Linux_Security_Modules</a></p>
</div>
<div class="sect4">
<h5 id="selinux"><a class="anchor" href="#selinux"></a><a class="link" href="#selinux">17.14.2.1. SELinux</a></h5>
<div class="paragraph">
<p>TODO get a hello world permission control working:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-linux \
  --config-fragment linux_config/selinux \
  --linux-build-id selinux \
;
./build-buildroot --config 'BR2_PACKAGE_REFPOLICY=y'
./run --enable-kvm --linux-build-id selinux</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/linux_config/selinux">linux_config/selinux</a></p>
</div>
<div class="paragraph">
<p>This builds:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>BR2_PACKAGE_REFPOLICY</code>, which includes a reference <code>/etc/selinux/config</code> policy: <a href="https://github.com/SELinuxProject/refpolicy" class="bare">https://github.com/SELinuxProject/refpolicy</a></p>
<div class="paragraph">
<p>refpolicy in turn depends on:</p>
</div>
</li>
<li>
<p><code>BR2_PACKAGE_SETOOLS</code>, which contains tools such as <code>getenforced</code>: <a href="https://github.com/SELinuxProject/setools" class="bare">https://github.com/SELinuxProject/setools</a></p>
<div class="paragraph">
<p>setools depends on:</p>
</div>
</li>
<li>
<p><code>BR2_PACKAGE_LIBSELINUX</code>, which is the backing userland library</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>After boot finishes, we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Starting auditd: mkdir: invalid option -- 'Z'</pre>
</div>
</div>
<div class="paragraph">
<p>which comes from <code>/etc/init.d/S01auditd</code>, because BusyBox' <code>mkdir</code> does not have the crazy <code>-Z</code> option like Ubuntu. That&#8217;s amazing!</p>
</div>
<div class="paragraph">
<p>The kernel logs contain:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>SELinux:  Initializing.</pre>
</div>
</div>
<div class="paragraph">
<p>Inside the guest we now have:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>getenforce</pre>
</div>
</div>
<div class="paragraph">
<p>which initially says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Disabled</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: if we try to enforce:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>setenforce 1</pre>
</div>
</div>
<div class="paragraph">
<p>it does not work and outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>setenforce: SELinux is disabled</pre>
</div>
</div>
<div class="paragraph">
<p>SELinux requires glibc as mentioned at: <a href="#libc-choice">Section 26.10, &#8220;libc choice&#8221;</a>.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="user-mode-linux"><a class="anchor" href="#user-mode-linux"></a><a class="link" href="#user-mode-linux">17.15. User mode Linux</a></h3>
<div class="paragraph">
<p>I once got <a href="https://en.wikipedia.org/wiki/User-mode_Linux">UML</a> running on a minimal Buildroot setup at: <a href="https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207" class="bare">https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207</a></p>
</div>
<div class="paragraph">
<p>But in part because it is dying, I didn&#8217;t spend much effort to integrate it into this repo, although it would be a good fit in principle, since it is essentially a virtualization method.</p>
</div>
<div class="paragraph">
<p>Maybe some brave soul will send a pull request one day.</p>
</div>
</div>
<div class="sect2">
<h3 id="uio"><a class="anchor" href="#uio"></a><a class="link" href="#uio">17.16. UIO</a></h3>
<div class="paragraph">
<p>UIO is a kernel subsystem that allows to do certain types of driver operations from userland.</p>
</div>
<div class="paragraph">
<p>This would be awesome to improve debuggability and safety of kernel modules.</p>
</div>
<div class="paragraph">
<p>VFIO looks like a newer and better UIO replacement, but there do not exist any examples of how to use it: <a href="https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver" class="bare">https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver</a></p>
</div>
<div class="paragraph">
<p>TODO get something interesting working. I currently don&#8217;t understand the behaviour very well.</p>
</div>
<div class="paragraph">
<p>TODO how to ACK interrupts? How to ensure that every interrupt gets handled separately?</p>
</div>
<div class="paragraph">
<p>TODO how to write to registers. Currently using <code>/dev/mem</code> and <code>lspci</code>.</p>
</div>
<div class="paragraph">
<p>This example should handle interrupts from userland and print a message to stdout:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./uio_read.sh</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what is the expected behaviour? I should have documented this when I wrote this stuff, and I&#8217;m that lazy right now that I&#8217;m in the middle of a refactor :-)</p>
</div>
<div class="paragraph">
<p>UIO interface in a nutshell:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>blocking read / poll: waits until interrupts</p>
</li>
<li>
<p><code>write</code>: call <code>irqcontrol</code> callback. Default: 0 or 1 to enable / disable interrupts.</p>
</li>
<li>
<p><code>mmap</code>: access device memory</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/kernel_modules/uio_read.c">userland/kernel_modules/uio_read.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/uio_read.sh">rootfs_overlay/lkmc/uio_read.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/15286772/userspace-vs-kernel-space-driver" class="bare">https://stackoverflow.com/questions/15286772/userspace-vs-kernel-space-driver</a></p>
</li>
<li>
<p><a href="https://01.org/linuxgraphics/gfx-docs/drm/driver-api/uio-howto.html" class="bare">https://01.org/linuxgraphics/gfx-docs/drm/driver-api/uio-howto.html</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/7986260/linux-interrupt-handling-in-user-space" class="bare">https://stackoverflow.com/questions/7986260/linux-interrupt-handling-in-user-space</a></p>
</li>
<li>
<p><a href="https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/" class="bare">https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/</a></p>
</li>
<li>
<p><a href="https://github.com/bmartini/zynq-axis/blob/65a3a448fda1f0ea4977adfba899eb487201853d/dev/axis.c" class="bare">https://github.com/bmartini/zynq-axis/blob/65a3a448fda1f0ea4977adfba899eb487201853d/dev/axis.c</a></p>
</li>
<li>
<p><a href="https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/" class="bare">https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/</a></p>
</li>
<li>
<p><a href="http://nairobi-embedded.org/uio_example.html" class="bare">http://nairobi-embedded.org/uio_example.html</a> that website has QEMU examples for everything as usual. The example has a kernel-side which creates the memory mappings and is used by the user.</p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver" class="bare">https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver</a></p>
</li>
<li>
<p>userland driver stability questions:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/8030758/getting-kernel-version-from-linux-kernel-module-at-runtime/45430233#45430233" class="bare">https://stackoverflow.com/questions/8030758/getting-kernel-version-from-linux-kernel-module-at-runtime/45430233#45430233</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681" class="bare">https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681</a></p>
</li>
<li>
<p><a href="https://liquidat.wordpress.com/2007/07/21/linux-kernel-2623-to-have-stable-userspace-driver-api/" class="bare">https://liquidat.wordpress.com/2007/07/21/linux-kernel-2623-to-have-stable-userspace-driver-api/</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-interactive-stuff"><a class="anchor" href="#linux-kernel-interactive-stuff"></a><a class="link" href="#linux-kernel-interactive-stuff">17.17. Linux kernel interactive stuff</a></h3>
<div class="sect3">
<h4 id="fbcon"><a class="anchor" href="#fbcon"></a><a class="link" href="#fbcon">17.17.1. Linux kernel console fun</a></h4>
<div class="paragraph">
<p>Requires <a href="#graphics">Graphics</a>.</p>
</div>
<div class="paragraph">
<p>You can also try those on the <code>Ctrl-Alt-F3</code> of your Ubuntu host, but it is much more fun inside a VM!</p>
</div>
<div class="paragraph">
<p>Stop the cursor from blinking:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 0 &gt; /sys/class/graphics/fbcon/cursor_blink</pre>
</div>
</div>
<div class="paragraph">
<p>Rotate the console 90 degrees! <a href="https://askubuntu.com/questions/237963/how-do-i-rotate-my-display-when-not-using-an-x-server" class="bare">https://askubuntu.com/questions/237963/how-do-i-rotate-my-display-when-not-using-an-x-server</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /sys/class/graphics/fbcon/rotate</pre>
</div>
</div>
<div class="paragraph">
<p>Relies on: <code>CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y</code>.</p>
</div>
<div class="paragraph">
<p>Documented under: <code>Documentation/fb/</code>.</p>
</div>
<div class="paragraph">
<p>TODO: font and keymap. Mentioned at: <a href="https://cmcenroe.me/2017/05/05/linux-console.html" class="bare">https://cmcenroe.me/2017/05/05/linux-console.html</a> and I think can be done with BusyBox <code>loadkmap</code> and <code>loadfont</code>, we just have to understand their formats, related:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/177024/remap-keyboard-on-the-linux-console" class="bare">https://unix.stackexchange.com/questions/177024/remap-keyboard-on-the-linux-console</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/194202/remapping-keys-system-wide-in-linux-not-just-in-x" class="bare">https://superuser.com/questions/194202/remapping-keys-system-wide-in-linux-not-just-in-x</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="linux-kernel-magic-keys"><a class="anchor" href="#linux-kernel-magic-keys"></a><a class="link" href="#linux-kernel-magic-keys">17.17.2. Linux kernel magic keys</a></h4>
<div class="paragraph">
<p>Requires <a href="#graphics">Graphics</a>.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s have some fun.</p>
</div>
<div class="paragraph">
<p>I think most are implemented under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>drivers/tty</pre>
</div>
</div>
<div class="paragraph">
<p>TODO find all.</p>
</div>
<div class="paragraph">
<p>Scroll up / down the terminal:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Shift-PgDown
Shift-PgUp</pre>
</div>
</div>
<div class="paragraph">
<p>Or inside <code>./qemu-monitor</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sendkey shift-pgup
sendkey shift-pgdown</pre>
</div>
</div>
<div class="sect4">
<h5 id="ctrl-alt-del"><a class="anchor" href="#ctrl-alt-del"></a><a class="link" href="#ctrl-alt-del">17.17.2.1. Ctrl Alt Del</a></h5>
<div class="paragraph">
<p>If you run in <a href="#qemu-graphic-mode">QEMU graphic mode</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>and then from the graphic window you enter the keys:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-Alt-Del</pre>
</div>
</div>
<div class="paragraph">
<p>then this runs the following command on the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/sbin/reboot</pre>
</div>
</div>
<div class="paragraph">
<p>This is enabled from our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/etc/inittab">rootfs_overlay/etc/inittab</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>::ctrlaltdel:/sbin/reboot</pre>
</div>
</div>
<div class="paragraph">
<p>This leads Linux to try to reboot, and QEMU shutdowns due to the <code>-no-reboot</code> option which we set by default for, see: <a href="#exit-emulator-on-panic">Section 17.6.1.3, &#8220;Exit emulator on panic&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Here is a minimal example of Ctrl Alt Del:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'init=/lkmc/linux/ctrl_alt_del.out' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/ctrl_alt_del.c">userland/linux/ctrl_alt_del.c</a></p>
</div>
<div class="paragraph">
<p>When you hit <code>Ctrl-Alt-Del</code> in the guest, our tiny init handles a <code>SIGINT</code> sent by the kernel and outputs to stdout:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cad</pre>
</div>
</div>
<div class="paragraph">
<p>To map between <code>man 2 reboot</code> and the uClibc <code>RB_*</code> magic constants see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar buildroot_build_build_dir)"/uclibc-*/include/sys/reboot.h"</pre>
</div>
</div>
<div class="paragraph">
<p>The procfs mechanism is documented at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less linux/Documentation/sysctl/kernel.txt</pre>
</div>
</div>
<div class="paragraph">
<p>which says:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>When the value in this file is 0, ctrl-alt-del is trapped and
sent to the init(1) program to handle a graceful restart.
When, however, the value is &gt; 0, Linux's reaction to a Vulcan
Nerve Pinch (tm) will be an immediate reboot, without even
syncing its dirty buffers.

Note: when a program (like dosemu) has the keyboard in 'raw'
mode, the ctrl-alt-del is intercepted by the program before it
ever reaches the kernel tty layer, and it's up to the program
to decide what to do with it.</pre>
</div>
</div>
<div class="paragraph">
<p>Under the hood, behaviour is controlled by the <code>reboot</code> syscall:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>man 2 reboot</pre>
</div>
</div>
<div class="paragraph">
<p><code>reboot</code> system calls can set either of the these behaviours for <code>Ctrl-Alt-Del</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>do a hard shutdown syscall. Set in uClibc C code with:</p>
<div class="literalblock">
<div class="content">
<pre>reboot(RB_ENABLE_CAD)</pre>
</div>
</div>
<div class="paragraph">
<p>or from procfs with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/kernel/ctrl-alt-del</pre>
</div>
</div>
<div class="paragraph">
<p>Done by BusyBox' <code>reboot -f</code>.</p>
</div>
</li>
<li>
<p>send a SIGINT to the init process. This is what BusyBox' init does, and it then execs the string set in <code>inittab</code>.</p>
<div class="paragraph">
<p>Set in uclibc C code with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>reboot(RB_DISABLE_CAD)</pre>
</div>
</div>
<div class="paragraph">
<p>or from procfs with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 0 &gt; /proc/sys/kernel/ctrl-alt-del</pre>
</div>
</div>
<div class="paragraph">
<p>Done by BusyBox' <code>reboot</code>.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>When a BusyBox init is with the signal, it prints the following lines:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot</pre>
</div>
</div>
<div class="paragraph">
<p>On busybox-1.29.2&#8217;s init at init/init.c we see how the kill signals are sent:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static void run_shutdown_and_kill_processes(void)
{
	/* Run everything to be run at "shutdown".  This is done _prior_
	 * to killing everything, in case people wish to use scripts to
	 * shut things down gracefully... */
	run_actions(SHUTDOWN);

	message(L_CONSOLE | L_LOG, "The system is going down NOW!");

	/* Send signals to every process _except_ pid 1 */
	kill(-1, SIGTERM);
	message(L_CONSOLE, "Sent SIG%s to all processes", "TERM");
	sync();
	sleep(1);

	kill(-1, SIGKILL);
	message(L_CONSOLE, "Sent SIG%s to all processes", "KILL");
	sync();
	/*sleep(1); - callers take care about making a pause */
}</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>run_shutdown_and_kill_processes</code> is called from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* The SIGPWR/SIGUSR[12]/SIGTERM handler */
static void halt_reboot_pwoff(int sig) NORETURN;
static void halt_reboot_pwoff(int sig)</pre>
</div>
</div>
<div class="paragraph">
<p>which also prints the final line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>	message(L_CONSOLE, "Requesting system %s", m);</pre>
</div>
</div>
<div class="paragraph">
<p>which is set as the signal handler via TODO.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/193652/does-linux-have-a-ctrlaltdel-equivalent/1324415#1324415" class="bare">https://superuser.com/questions/193652/does-linux-have-a-ctrlaltdel-equivalent/1324415#1324415</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/42573/meaning-and-commands-for-ctrlaltdel/444969#444969" class="bare">https://unix.stackexchange.com/questions/42573/meaning-and-commands-for-ctrlaltdel/444969#444969</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="sysrq"><a class="anchor" href="#sysrq"></a><a class="link" href="#sysrq">17.17.2.2. SysRq</a></h5>
<div class="paragraph">
<p>We cannot test these actual shortcuts on QEMU since the host captures them at a lower level, but from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor</pre>
</div>
</div>
<div class="paragraph">
<p>we can for example crash the system with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sendkey alt-sysrq-c</pre>
</div>
</div>
<div class="paragraph">
<p>Same but boring because no magic key:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo c &gt; /proc/sysrq-trigger</pre>
</div>
</div>
<div class="paragraph">
<p>Implemented in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>drivers/tty/sysrq.c</pre>
</div>
</div>
<div class="paragraph">
<p>On your host, on modern systems that don&#8217;t have the <code>SysRq</code> key you can do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Alt-PrtSc-space</pre>
</div>
</div>
<div class="paragraph">
<p>which prints a message to <code>dmesg</code> of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)</pre>
</div>
</div>
<div class="paragraph">
<p>Individual SysRq can be enabled or disabled with the bitmask:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/proc/sys/kernel/sysrq</pre>
</div>
</div>
<div class="paragraph">
<p>The bitmask is documented at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less linux/Documentation/admin-guide/sysrq.rst</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://en.wikipedia.org/wiki/Magic_SysRq_key" class="bare">https://en.wikipedia.org/wiki/Magic_SysRq_key</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="tty"><a class="anchor" href="#tty"></a><a class="link" href="#tty">17.17.3. TTY</a></h4>
<div class="paragraph">
<p>In order to play with TTYs, do this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '
tty2::respawn:/sbin/getty -n -L -l /lkmc/loginroot.sh tty2 0 vt100
tty3::respawn:-/bin/sh
tty4::respawn:/sbin/getty 0 tty4
tty63::respawn:-/bin/sh
::respawn:/sbin/getty -L ttyS0 0 vt100
::respawn:/sbin/getty -L ttyS1 0 vt100
::respawn:/sbin/getty -L ttyS2 0 vt100
# Leave one serial empty.
#::respawn:/sbin/getty -L ttyS3 0 vt100
' &gt;&gt; rootfs_overlay/etc/inittab
./build-buildroot
./run --graphic -- \
  -serial telnet::1235,server,nowait \
  -serial vc:800x600 \
  -serial telnet::1236,server,nowait \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and on a second shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>telnet localhost 1235</pre>
</div>
</div>
<div class="paragraph">
<p>We don&#8217;t add more TTYs by default because it would spawn more processes, even if we use <code>askfirst</code> instead of <code>respawn</code>.</p>
</div>
<div class="paragraph">
<p>On the GUI, switch TTYs with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>Alt-Left</code> or <code>Alt-Right:</code> go to previous / next populated <code>/dev/ttyN</code> TTY. Skips over empty TTYs.</p>
</li>
<li>
<p><code>Alt-Fn</code>: go to the nth TTY. If it is  not populated, don&#8217;t go there.</p>
</li>
<li>
<p><code>chvt &lt;n&gt;</code>: go to the n-th virtual TTY, even if it is empty: <a href="https://superuser.com/questions/33065/console-commands-to-change-virtual-ttys-in-linux-and-openbsd" class="bare">https://superuser.com/questions/33065/console-commands-to-change-virtual-ttys-in-linux-and-openbsd</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>You can also test this on most hosts such as Ubuntu 18.04, except that when in the GUI, you must use <code>Ctrl-Alt-Fx</code> to switch to another terminal.</p>
</div>
<div class="paragraph">
<p>Next, we also have the following shells running on the serial ports, hit enter to activate them:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/dev/ttyS0</code>: first shell that was used to run QEMU, corresponds to QEMU&#8217;s <code>-serial mon:stdio</code>.</p>
<div class="paragraph">
<p>It would also work if we used <code>-serial stdio</code>, but:</p>
</div>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p><code>Ctrl-C</code> would kill QEMU instead of going to the guest</p>
</li>
<li>
<p><code>Ctrl-A C</code> wouldn&#8217;t open the QEMU console there</p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>see also: <a href="https://stackoverflow.com/questions/49716931/how-to-run-qemu-with-nographic-and-monitor-but-still-be-able-to-send-ctrlc-to" class="bare">https://stackoverflow.com/questions/49716931/how-to-run-qemu-with-nographic-and-monitor-but-still-be-able-to-send-ctrlc-to</a></p>
</div>
</li>
<li>
<p><code>/dev/ttyS1</code>: second shell running <code>telnet</code></p>
</li>
<li>
<p><code>/dev/ttyS2</code>: go on the GUI and enter <code>Ctrl-Alt-2</code>, corresponds to QEMU&#8217;s <code>-serial vc</code>. Go back to the main console with <code>Ctrl-Alt-1</code>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>although we cannot change between terminals from there.</p>
</div>
<div class="paragraph">
<p>Each populated TTY contains a "shell":</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>-/bin/sh</code>: goes directly into an <code>sh</code> without a login prompt.</p>
<div class="paragraph">
<p>The trailing dash <code>-</code> can be used on any command. It makes the command that follows take over the TTY, which is what we typically want for interactive shells: <a href="https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using" class="bare">https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using</a></p>
</div>
<div class="paragraph">
<p>The <code>getty</code> executable however also does this operation and therefore dispenses the <code>-</code>.</p>
</div>
</li>
<li>
<p><code>/sbin/getty</code> asks for password, and then gives you an <code>sh</code></p>
<div class="paragraph">
<p>We can overcome the password prompt with the <code>-l /lkmc/loginroot.sh</code> technique explained at: <a href="https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using" class="bare">https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using</a> but I don&#8217;t see any advantage over <code>-/bin/sh</code> currently.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Identify the current TTY with the command:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>tty</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://unix.stackexchange.com/questions/270272/how-to-get-the-tty-in-which-bash-is-running/270372" class="bare">https://unix.stackexchange.com/questions/270272/how-to-get-the-tty-in-which-bash-is-running/270372</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/187319/how-to-get-the-real-name-of-the-controlling-terminal" class="bare">https://unix.stackexchange.com/questions/187319/how-to-get-the-real-name-of-the-controlling-terminal</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/77796/how-to-get-the-current-terminal-name" class="bare">https://unix.stackexchange.com/questions/77796/how-to-get-the-current-terminal-name</a></p>
</li>
<li>
<p><a href="https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using" class="bare">https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This outputs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>/dev/console</code> for the initial GUI terminal. But I think it is the same as <code>/dev/tty1</code>, because if I try to do</p>
<div class="literalblock">
<div class="content">
<pre>tty1::respawn:-/bin/sh</pre>
</div>
</div>
<div class="paragraph">
<p>it makes the terminal go crazy, as if multiple processes are randomly eating up the characters.</p>
</div>
</li>
<li>
<p><code>/dev/ttyN</code> for the other graphic TTYs. Note that there are only 63 available ones, from <code>/dev/tty1</code> to <code>/dev/tty63</code> (<code>/dev/tty0</code> is the current one): <a href="https://superuser.com/questions/449781/why-is-there-so-many-linux-dev-tty" class="bare">https://superuser.com/questions/449781/why-is-there-so-many-linux-dev-tty</a>. I think this is determined by:</p>
<div class="literalblock">
<div class="content">
<pre>#define MAX_NR_CONSOLES 63</pre>
</div>
</div>
<div class="paragraph">
<p>in <code>linux/include/uapi/linux/vt.h</code>.</p>
</div>
</li>
<li>
<p><code>/dev/ttySN</code> for the text shells.</p>
<div class="paragraph">
<p>These are Serial ports, see this to understand what those represent physically: <a href="https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux/367882#367882" class="bare">https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux/367882#367882</a></p>
</div>
<div class="paragraph">
<p>There are only 4 serial ports, I think this is determined by QEMU. TODO check.</p>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/16706423/two-instances-of-busybox-on-separate-serial-lines-ttysn" class="bare">https://stackoverflow.com/questions/16706423/two-instances-of-busybox-on-separate-serial-lines-ttysn</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Get the TTY in bulk for all processes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./psa.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/psa.sh">rootfs_overlay/lkmc/psa.sh</a>.</p>
</div>
<div class="paragraph">
<p>The TTY appears under the <code>TT</code> section, which is enabled by <code>-o tty</code>. This shows the TTY device number, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>4,1</pre>
</div>
</div>
<div class="paragraph">
<p>and we can then confirm it with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls -l /dev/tty1</pre>
</div>
</div>
<div class="paragraph">
<p>Next try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod kthread.ko</pre>
</div>
</div>
<div class="paragraph">
<p>and switch between virtual terminals, to understand that the dmesg goes to whatever current virtual terminal you are on, but not the others, and not to the serial terminals.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://serverfault.com/questions/119736/how-to-enable-multiple-virtual-consoles-on-linux" class="bare">https://serverfault.com/questions/119736/how-to-enable-multiple-virtual-consoles-on-linux</a></p>
</li>
<li>
<p><a href="https://github.com/mirror/busybox/blob/1_28_3/examples/inittab#L60" class="bare">https://github.com/mirror/busybox/blob/1_28_3/examples/inittab#L60</a></p>
</li>
<li>
<p><a href="http://web.archive.org/web/20180117124612/http://nairobi-embedded.org/qemu_serial_port_system_console.html" class="bare">http://web.archive.org/web/20180117124612/http://nairobi-embedded.org/qemu_serial_port_system_console.html</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="start-a-getty-from-outside-of-init"><a class="anchor" href="#start-a-getty-from-outside-of-init"></a><a class="link" href="#start-a-getty-from-outside-of-init">17.17.3.1. Start a getty from outside of init</a></h5>
<div class="paragraph">
<p>TODO: <a href="https://unix.stackexchange.com/questions/196704/getty-start-from-command-line" class="bare">https://unix.stackexchange.com/questions/196704/getty-start-from-command-line</a></p>
</div>
<div class="paragraph">
<p>TODO: how to place an <code>sh</code> directly on a TTY as well without <code>getty</code>?</p>
</div>
<div class="paragraph">
<p>If I try the exact same command that the <code>inittab</code> is doing from a regular shell after boot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/sbin/getty 0 tty1</pre>
</div>
</div>
<div class="paragraph">
<p>it fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>getty: setsid: Operation not permitted</pre>
</div>
</div>
<div class="paragraph">
<p>The following however works:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval 'getty 0 tty1 &amp; getty 0 tty2 &amp; getty 0 tty3 &amp; sleep 99999999' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>presumably because it is being called from <code>init</code> directly?</p>
</div>
<div class="paragraph">
<p>Outcome: <code>Alt-Right</code> cycles between three TTYs, <code>tty1</code> being the default one that appears under the boot messages.</p>
</div>
<div class="paragraph">
<p><code>man 2 setsid</code> says that there is only one failure possibility:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>EPERM  The process group ID of any process equals the PID of the calling process.  Thus, in particular, setsid() fails if the calling process is already a process group leader.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>We can get some visibility into it to try and solve the problem with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./psa.sh</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="console-kernel-boot-parameter"><a class="anchor" href="#console-kernel-boot-parameter"></a><a class="link" href="#console-kernel-boot-parameter">17.17.3.2. console kernel boot parameter</a></h5>
<div class="paragraph">
<p>Take the command described at <a href="#tty">TTY</a> and try adding the following:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>-e 'console=tty7'</code>: boot messages still show on <code>/dev/tty1</code> (TODO how to change that?), but we don&#8217;t get a shell at the end of boot there.</p>
<div class="paragraph">
<p>Instead, the shell appears on <code>/dev/tty7</code>.</p>
</div>
</li>
<li>
<p><code>-e 'console=tty2'</code> like <code>/dev/tty7</code>, but <code>/dev/tty2</code> is broken, because we have two shells there:</p>
<div class="ulist">
<ul>
<li>
<p>one due to the <code>::respawn:-/bin/sh</code> entry which uses whatever <code>console</code> points to</p>
</li>
<li>
<p>another one due to the <code>tty2::respawn:/sbin/getty</code> entry we added</p>
</li>
</ul>
</div>
</li>
<li>
<p><code>-e 'console=ttyS0'</code> much like <code>tty2</code>, but messages show only on serial, and the terminal is broken due to having multiple shells on it</p>
</li>
<li>
<p><code>-e 'console=tty1 console=ttyS0'</code>: boot messages show on both <code>tty1</code> and <code>ttyS0</code>, but only <code>S0</code> gets a shell because it came last</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="config-logo"><a class="anchor" href="#config-logo"></a><a class="link" href="#config-logo">17.17.4. CONFIG_LOGO</a></h4>
<div class="paragraph">
<p>If you run in <a href="#graphics">Graphics</a>, then you get a Penguin image for <a href="#number-of-cores">every core</a> above the console! <a href="https://askubuntu.com/questions/80938/is-it-possible-to-get-the-tux-logo-on-the-text-based-boot" class="bare">https://askubuntu.com/questions/80938/is-it-possible-to-get-the-tux-logo-on-the-text-based-boot</a></p>
</div>
<div class="paragraph">
<p>This is due to the <a href="https://github.com/torvalds/linux/blob/v4.17/drivers/video/logo/Kconfig#L5"><code>CONFIG_LOGO=y</code></a> option which we enable by default.</p>
</div>
<div class="paragraph">
<p><code>reset</code> on the terminal then kills the poor penguins.</p>
</div>
<div class="paragraph">
<p>When <code>CONFIG_LOGO=y</code> is set, the logo can be disabled at boot with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --kernel-cli 'logo.nologo'</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/39872463/how-can-i-disable-the-startup-penguins-and-boot-text-on-linaro-ubuntu" class="bare">https://stackoverflow.com/questions/39872463/how-can-i-disable-the-startup-penguins-and-boot-text-on-linaro-ubuntu</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/332198/centos-remove-penguin-logo-at-startup" class="bare">https://unix.stackexchange.com/questions/332198/centos-remove-penguin-logo-at-startup</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Looks like a recompile is needed to modify the image&#8230;&#8203;</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/736423/changing-kernel-bootsplash-image" class="bare">https://superuser.com/questions/736423/changing-kernel-bootsplash-image</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/153975/how-to-change-boot-logo-in-linux-mint" class="bare">https://unix.stackexchange.com/questions/153975/how-to-change-boot-logo-in-linux-mint</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="drm"><a class="anchor" href="#drm"></a><a class="link" href="#drm">17.18. DRM</a></h3>
<div class="paragraph">
<p>DRM / DRI is the new interface that supersedes <code>fbdev</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_LIBDRM=y'
./build-userland --package libdrm -- userland/libs/libdrm/modeset.c
./run --eval-after './libs/libdrm/modeset.out' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/libdrm/modeset.c">userland/libs/libdrm/modeset.c</a></p>
</div>
<div class="paragraph">
<p>Outcome: for a few seconds, the screen that contains the terminal gets taken over by changing colors of the rainbow.</p>
</div>
<div class="paragraph">
<p>TODO not working for <code>aarch64</code>, it takes over the screen for a few seconds and the kernel messages disappear, but the screen stays black all the time.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_LIBDRM=y'
./build-userland --package libdrm
./build-buildroot
./run --eval-after './libs/libdrm/modeset.out' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p><a href="#kmscube">kmscube</a> however worked, which means that it must be a bug with this demo?</p>
</div>
<div class="paragraph">
<p>We set <code>CONFIG_DRM=y</code> on our default kernel configuration, and it creates one device file for each display:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># ls -l /dev/dri
total 0
crw-------    1 root     root      226,   0 May 28 09:41 card0
# grep 226 /proc/devices
226 drm
# ls /sys/module/drm /sys/module/drm_kms_helper/</pre>
</div>
</div>
<div class="paragraph">
<p>Try creating new displays:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --graphic -- -device virtio-gpu-pci</pre>
</div>
</div>
<div class="paragraph">
<p>to see multiple <code>/dev/dri/cardN</code>, and then use a different display with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './libs/libdrm/modeset.out' --graphic</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://dri.freedesktop.org/wiki/DRM/" class="bare">https://dri.freedesktop.org/wiki/DRM/</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure" class="bare">https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Direct_Rendering_Manager" class="bare">https://en.wikipedia.org/wiki/Direct_Rendering_Manager</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Mode_setting" class="bare">https://en.wikipedia.org/wiki/Mode_setting</a> KMS</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/93e383902ebcc03d8a7ac0d65961c0e62af9612b">93e383902ebcc03d8a7ac0d65961c0e62af9612b</a></p>
</div>
<div class="sect3">
<h4 id="kmscube"><a class="anchor" href="#kmscube"></a><a class="link" href="#kmscube">17.18.1. kmscube</a></h4>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config-fragment buildroot_config/kmscube</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: a colored spinning cube coded in OpenGL + EGL takes over your display and spins forever: <a href="https://www.youtube.com/watch?v=CqgJMgfxjsk" class="bare">https://www.youtube.com/watch?v=CqgJMgfxjsk</a></p>
</div>
<div class="paragraph">
<p>It is a bit amusing to see OpenGL running outside of a window manager window like that: <a href="https://stackoverflow.com/questions/3804065/using-opengl-without-a-window-manager-in-linux/50669152#50669152" class="bare">https://stackoverflow.com/questions/3804065/using-opengl-without-a-window-manager-in-linux/50669152#50669152</a></p>
</div>
<div class="paragraph">
<p>TODO: it is very slow, about 1FPS. I tried Buildroot master ad684c20d146b220dd04a85dbf2533c69ec8ee52 with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>make qemu_x86_64_defconfig
printf "
BR2_CCACHE=y
BR2_PACKAGE_HOST_QEMU=y
BR2_PACKAGE_HOST_QEMU_LINUX_USER_MODE=n
BR2_PACKAGE_HOST_QEMU_SYSTEM_MODE=y
BR2_PACKAGE_HOST_QEMU_VDE2=y
BR2_PACKAGE_KMSCUBE=y
BR2_PACKAGE_MESA3D=y
BR2_PACKAGE_MESA3D_DRI_DRIVER_SWRAST=y
BR2_PACKAGE_MESA3D_OPENGL_EGL=y
BR2_PACKAGE_MESA3D_OPENGL_ES=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
" &gt;&gt; .config</pre>
</div>
</div>
<div class="paragraph">
<p>and the FPS was much better, I estimate something like 15FPS.</p>
</div>
<div class="paragraph">
<p>On Ubuntu 18.04 with NVIDIA proprietary drivers:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get instll kmscube
kmscube</pre>
</div>
</div>
<div class="paragraph">
<p>fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>drmModeGetResources failed: Invalid argument
failed to initialize legacy DRM</pre>
</div>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/robclark/kmscube/issues/12" class="bare">https://github.com/robclark/kmscube/issues/12</a> and:</p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/26920835/can-egl-application-run-in-console-mode/26921287#26921287" class="bare">https://stackoverflow.com/questions/26920835/can-egl-application-run-in-console-mode/26921287#26921287</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/3804065/how-to-use-opengl-without-a-window-manager-in-linux/50669152#50669152" class="bare">https://stackoverflow.com/questions/3804065/how-to-use-opengl-without-a-window-manager-in-linux/50669152#50669152</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/2903771275372ccfecc2b025edbb0d04c4016930">2903771275372ccfecc2b025edbb0d04c4016930</a></p>
</div>
</div>
<div class="sect3">
<h4 id="kmscon"><a class="anchor" href="#kmscon"></a><a class="link" href="#kmscon">17.18.2. kmscon</a></h4>
<div class="paragraph">
<p>TODO get working.</p>
</div>
<div class="paragraph">
<p>Implements a console for <a href="#drm">DRM</a>.</p>
</div>
<div class="paragraph">
<p>The Linux kernel has a built-in fbdev console called <a href="#fbcon">Linux kernel console fun</a> but not for <a href="#drm">DRM</a> it seems.</p>
</div>
<div class="paragraph">
<p>The upstream project seems dead with last commit in 2014: <a href="https://www.freedesktop.org/wiki/Software/kmscon/" class="bare">https://www.freedesktop.org/wiki/Software/kmscon/</a></p>
</div>
<div class="paragraph">
<p>Build failed in Ubuntu 18.04 with: <a href="https://github.com/dvdhrm/kmscon/issues/131" class="bare">https://github.com/dvdhrm/kmscon/issues/131</a> but this fork compiled but didn&#8217;t run on host: <a href="https://github.com/Aetf/kmscon/issues/2#issuecomment-392484043" class="bare">https://github.com/Aetf/kmscon/issues/2#issuecomment-392484043</a></p>
</div>
<div class="paragraph">
<p>Haven&#8217;t tested the fork on QEMU too much insanity.</p>
</div>
</div>
<div class="sect3">
<h4 id="libdri2"><a class="anchor" href="#libdri2"></a><a class="link" href="#libdri2">17.18.3. libdri2</a></h4>
<div class="paragraph">
<p>TODO get working.</p>
</div>
<div class="paragraph">
<p>Looks like a more raw alternative to libdrm:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKABE_LIBDRI2=y'
wget \
  -O "$(./getvar userland_source_dir)/dri2test.c" \
  https://raw.githubusercontent.com/robclark/libdri2/master/test/dri2test.c \
;
./build-userland</pre>
</div>
</div>
<div class="paragraph">
<p>but then I noticed that that example requires multiple files, and I don&#8217;t feel like integrating it into our build.</p>
</div>
<div class="paragraph">
<p>When I build it on Ubuntu 18.04 host, it does not generate any executable, so I&#8217;m confused.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-testing"><a class="anchor" href="#linux-kernel-testing"></a><a class="link" href="#linux-kernel-testing">17.19. Linux kernel testing</a></h3>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested" class="bare">https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested</a></p>
</div>
<div class="sect3">
<h4 id="linux-test-project"><a class="anchor" href="#linux-test-project"></a><a class="link" href="#linux-test-project">17.19.1. Linux Test Project</a></h4>
<div class="paragraph">
<p><a href="https://github.com/linux-test-project/ltp" class="bare">https://github.com/linux-test-project/ltp</a></p>
</div>
<div class="paragraph">
<p>Tests a lot of Linux and POSIX userland visible interfaces.</p>
</div>
<div class="paragraph">
<p>Buildroot already has a package, so it is trivial to build it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_LTP_TESTSUITE=y'</pre>
</div>
</div>
<div class="paragraph">
<p>So now let&#8217;s try and see if the <code>exit</code> system call is working:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/usr/lib/ltp-testsuite/testcases/bin/exit01</pre>
</div>
</div>
<div class="paragraph">
<p>which gives successful output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>exit01      1  TPASS  :  exit() test PASSED</pre>
</div>
</div>
<div class="paragraph">
<p>and has source code at: <a href="https://github.com/linux-test-project/ltp/blob/20190115/testcases/kernel/syscalls/exit/exit01.c" class="bare">https://github.com/linux-test-project/ltp/blob/20190115/testcases/kernel/syscalls/exit/exit01.c</a></p>
</div>
<div class="paragraph">
<p>Besides testing any kernel modifications you make, LTP can also be used to the system call implementation of <a href="#user-mode-simulation">User mode simulation</a> as shown at <a href="#user-mode-buildroot-executables">User mode Buildroot executables</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland "$(./getvar buildroot_target_dir)/usr/lib/ltp-testsuite/testcases/bin/exit01"</pre>
</div>
</div>
<div class="paragraph">
<p>Tested at: 287c83f3f99db8c1ff9bbc85a79576da6a78e986 + 1.</p>
</div>
</div>
<div class="sect3">
<h4 id="stress"><a class="anchor" href="#stress"></a><a class="link" href="#stress">17.19.2. stress</a></h4>
<div class="paragraph">
<p><a href="#posix">POSIX</a> userland stress. Two versions:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --config 'BR2_PACKAGE_STRESS=y' \
  --config 'BR2_PACKAGE_STRESS_NG=y' \
;</pre>
</div>
</div>
<div class="paragraph">
<p><code>STRESS_NG</code> is likely the best, but it requires glibc, see: <a href="#libc-choice">Section 26.10, &#8220;libc choice&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Websites:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://people.seas.harvard.edu/~apw/stress/" class="bare">https://people.seas.harvard.edu/~apw/stress/</a></p>
</li>
<li>
<p><a href="https://github.com/ColinIanKing/stress-ng" class="bare">https://github.com/ColinIanKing/stress-ng</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>stress</code> usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>stress --help
stress -c 16 &amp;
ps</pre>
</div>
</div>
<div class="paragraph">
<p>and notice how 16 threads were created in addition to a parent worker thread.</p>
</div>
<div class="paragraph">
<p>It just runs forever, so kill it when you get tired:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>kill %1</pre>
</div>
</div>
<div class="paragraph">
<p><code>stress -c 1 -t 1</code> makes gem5 irresponsive for a very long time.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-kernel-build-system"><a class="anchor" href="#linux-kernel-build-system"></a><a class="link" href="#linux-kernel-build-system">17.20. Linux kernel build system</a></h3>
<div class="sect3">
<h4 id="vmlinux-vs-bzimage-vs-zimage-vs-image"><a class="anchor" href="#vmlinux-vs-bzimage-vs-zimage-vs-image"></a><a class="link" href="#vmlinux-vs-bzimage-vs-zimage-vs-image">17.20.1. vmlinux vs bzImage vs zImage vs Image</a></h4>
<div class="paragraph">
<p>Between all archs on QEMU and gem5 we touch all of those kernel built output files.</p>
</div>
<div class="paragraph">
<p>We are trying to maintain a description of each at: <a href="https://unix.stackexchange.com/questions/5518/what-is-the-difference-between-the-following-kernel-makefile-terms-vmlinux-vml/482978#482978" class="bare">https://unix.stackexchange.com/questions/5518/what-is-the-difference-between-the-following-kernel-makefile-terms-vmlinux-vml/482978#482978</a></p>
</div>
<div class="paragraph">
<p>QEMU does not seem able to boot ELF files like <code>vmlinux</code>: <a href="https://superuser.com/questions/1376944/can-qemu-boot-linux-from-vmlinux-instead-of-bzimage" class="bare">https://superuser.com/questions/1376944/can-qemu-boot-linux-from-vmlinux-instead-of-bzimage</a></p>
</div>
<div class="paragraph">
<p>Converting <code>arch/*</code> images to <code>vmlinux</code> is possible in theory x86 with <a href="https://github.com/torvalds/linux/blob/v5.1/scripts/extract-vmlinux"><code>extract-vmlinux</code></a> but we didn&#8217;t get any gem5 boots working from images generated like that for some reason, see: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="virtio"><a class="anchor" href="#virtio"></a><a class="link" href="#virtio">17.21. Virtio</a></h3>
<div class="paragraph">
<p><a href="https://www.linux-kvm.org/page/Virtio" class="bare">https://www.linux-kvm.org/page/Virtio</a></p>
</div>
<div class="paragraph">
<p>Virtio is an interface that guest machines can use to efficiently use resources from host machines.</p>
</div>
<div class="paragraph">
<p>The types of resources it supports are for disks and networking hardware.</p>
</div>
<div class="paragraph">
<p>This interface is not like the real interface used by the host to read from real disks and network devices.</p>
</div>
<div class="paragraph">
<p>Rather, it is a simplified interface, that makes those operations simpler and faster since guest and host work together knowing that this is an emulation use case.</p>
</div>
</div>
<div class="sect2">
<h3 id="kernel-modules"><a class="anchor" href="#kernel-modules"></a><a class="link" href="#kernel-modules">17.22. Kernel modules</a></h3>
<div class="sect3">
<h4 id="dump-regs"><a class="anchor" href="#dump-regs"></a><a class="link" href="#dump-regs">17.22.1. dump_regs</a></h4>
<div class="paragraph">
<p>The following kernel modules and <a href="#baremetal">Baremetal</a> executables dump and disassemble various registers which cannot be observed from userland (usually "system registers", "control registers"):</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/dump_regs.c">kernel_modules/dump_regs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/dump_regs.c">userland/arch/arm/dump_regs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/dump_regs.c">userland/arch/aarch64/dump_regs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/dump_regs.c">baremetal/arch/arm/dump_regs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/dump_regs.c">baremetal/arch/aarch64/dump_regs.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Some of those programs are using:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/aarch64_dump_regs.h">lkmc/aarch64_dump_regs.h</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Alternatively, you can also get their value from inside <a href="#gdb">GDB step debug</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info registers all</pre>
</div>
</div>
<div class="paragraph">
<p>or the short version:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i r a</pre>
</div>
</div>
<div class="paragraph">
<p>or to get just specific registers, e.g. just ARMv8&#8217;s SCTLR:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i r SCTLR</pre>
</div>
</div>
<div class="paragraph">
<p>but it is sometimes just more convenient to run an executable to get the registers at the point of interest.</p>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/5429137/how-to-print-register-values-in-gdb/31340294#31340294" class="bare">https://stackoverflow.com/questions/5429137/how-to-print-register-values-in-gdb/31340294#31340294</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/24169614/how-to-show-all-x86-control-registers-when-debugging-the-linux-kernel-in-gdb-thr/59311764#59311764" class="bare">https://stackoverflow.com/questions/24169614/how-to-show-all-x86-control-registers-when-debugging-the-linux-kernel-in-gdb-thr/59311764#59311764</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="freebsd"><a class="anchor" href="#freebsd"></a><a class="link" href="#freebsd">18. FreeBSD</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/FreeBSD" class="bare">https://en.wikipedia.org/wiki/FreeBSD</a></p>
</div>
<div class="paragraph">
<p>Prebuilt on Ubuntu 20.04 worked: <a href="https://stackoverflow.com/questions/49656395/how-to-boot-freebsd-image-under-qemu/64027161#64027161" class="bare">https://stackoverflow.com/questions/49656395/how-to-boot-freebsd-image-under-qemu/64027161#64027161</a></p>
</div>
<div class="paragraph">
<p>TODO minimal build + boot on QEMU example anywhere???</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="rtos"><a class="anchor" href="#rtos"></a><a class="link" href="#rtos">19. RTOS</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Real-time_operating_system" class="bare">https://en.wikipedia.org/wiki/Real-time_operating_system</a></p>
</div>
<div class="sect2">
<h3 id="zephyr"><a class="anchor" href="#zephyr"></a><a class="link" href="#zephyr">19.1. Zephyr</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Zephyr_(operating_system" class="bare">https://en.wikipedia.org/wiki/Zephyr_(operating_system</a>)</p>
</div>
<div class="paragraph">
<p>Zephyr is an RTOS that has <a href="#posix">POSIX</a> support. I think it works much like our <a href="#baremetal-setup">Baremetal setup</a> which uses Newlib and generates individual ELF files that contain both our C program&#8217;s code, and the Zephyr libraries.</p>
</div>
<div class="paragraph">
<p>TODO get a hello world working, and then consider further integration in this repo, e.g. being able to run all C userland content on it.</p>
</div>
<div class="paragraph">
<p>TODO: Cortex-A CPUs are not currently supported, there are some <code>qemu_cortex_m0</code> boards, but can&#8217;t find a QEMU Cortex-A. There is an x86_64 qemu board, but we don&#8217;t currently have an <a href="#about-the-baremetal-setup">x86 baremetal toolchain</a>. For this reason, we won&#8217;t touch this further for now.</p>
</div>
<div class="paragraph">
<p>However, unlike Newlib, Zephyr must be setting up a simple pre-main runtime to be able to handle threads.</p>
</div>
<div class="paragraph">
<p>Failed attempt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># https://askubuntu.com/questions/952429/is-there-a-good-ppa-for-cmake-backports
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2&gt;/dev/null | sudo apt-key add -
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic-rc main'
sudo apt-get update
sudo apt-get install cmake
git clone https://github.com/zephyrproject-rtos/zephyr
pip3 install --user -U west packaging
cd zephyr
git checkout v1.14.1
west init zephyrproject
west update
export ZEPHYR_TOOLCHAIN_VARIANT=xtools
export XTOOLS_TOOLCHAIN_PATH="$(pwd)/out/crosstool-ng/build/default/install/aarch64/bin/"
source zephyr-env.sh
west build -b qemu_aarch64 samples/hello_world</pre>
</div>
</div>
<div class="paragraph">
<p>The build system of that project is a bit excessive / wonky. You need an edge CMake not present in Ubuntu 18.04, which I don&#8217;t want to install right now, and it uses the weird custom <code>west</code> build tool frontend.</p>
</div>
</div>
<div class="sect2">
<h3 id="arm-mbed"><a class="anchor" href="#arm-mbed"></a><a class="link" href="#arm-mbed">19.2. ARM Mbed</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Mbed" class="bare">https://en.wikipedia.org/wiki/Mbed</a></p>
</div>
<div class="paragraph">
<p>TODO minimal setup to run it on QEMU? Possible?</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="xen"><a class="anchor" href="#xen"></a><a class="link" href="#xen">20. Xen</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Xen" class="bare">https://en.wikipedia.org/wiki/Xen</a></p>
</div>
<div class="paragraph">
<p>TODO: get prototype working and then properly integrate:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-xen</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-xen">build-xen</a></p>
</div>
<div class="paragraph">
<p>This script attempts to build Xen for aarch64 and feed it into QEMU through <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/submodules/boot-wrapper-aarch64">submodules/boot-wrapper-aarch64</a></p>
</div>
<div class="paragraph">
<p>TODO: other archs not yet attempted.</p>
</div>
<div class="paragraph">
<p>The current bad behaviour is that it prints just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Boot-wrapper v0.2</pre>
</div>
</div>
<div class="paragraph">
<p>and nothing else.</p>
</div>
<div class="paragraph">
<p>We will also need <code>CONFIG_XEN=y</code> on the Linux kernel, but first Xen should print some Xen messages before the kernel is ever reached.</p>
</div>
<div class="paragraph">
<p>If we pass to QEMU the xen image directly instead of the boot wrapper one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-kernel ../xen/xen/xen</pre>
</div>
</div>
<div class="paragraph">
<p>then Xen messages do show up! So it seems that the configuration failure lies in the boot wrapper itself rather than Xen.</p>
</div>
<div class="paragraph">
<p>Maybe it is also possible to run Xen directly like this: QEMU can already load multiple images at different memory locations with the generic loader: <a href="https://github.com/qemu/qemu/blob/master/docs/generic-loader.txt" class="bare">https://github.com/qemu/qemu/blob/master/docs/generic-loader.txt</a> which looks something along:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-kernel file1.elf -device loader,file=file2.elf</pre>
</div>
</div>
<div class="paragraph">
<p>so as long as we craft the correct DTB and feed it into Xen so that it can see the kernel, it should work. TODO does QEMU support patching the auto-generated DTB with pre-generated options? In the worst case we can just dump it hand hack it up though with <code>-machine dumpdtb</code>, see: <a href="#device-tree-emulator-generation">Section 9.4, &#8220;Device tree emulator generation&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>this attempt was based on: <a href="https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels" class="bare">https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels</a> which is the documentation for the ARM Fast Models closed source simulators.</p>
</li>
<li>
<p><a href="https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64" class="bare">https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64</a> this is the only QEMU aarch64 Xen page on the web. It uses the Ubuntu aarc64 image, which has EDK2.</p>
<div class="paragraph">
<p>I however see no joy on blobs. Buildroot does not seem to support EDK 2.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Link on readme <a href="https://stackoverflow.com/questions/49348453/xen-on-qemu-with-arm64-architecture" class="bare">https://stackoverflow.com/questions/49348453/xen-on-qemu-with-arm64-architecture</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="u-boot"><a class="anchor" href="#u-boot"></a><a class="link" href="#u-boot">21. U-Boot</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Das_U-Boot" class="bare">https://en.wikipedia.org/wiki/Das_U-Boot</a></p>
</div>
<div class="paragraph">
<p>U-Boot is a popular bootloader.</p>
</div>
<div class="paragraph">
<p>It can read disk filesystems, and Buildroot supports it, so we could in theory put it into memory, and let it find a kernel image from the root filesystem and boot that, but I didn&#8217;t manage to get it working yet: <a href="https://stackoverflow.com/questions/58028789/how-to-boot-linux-aarch64-with-u-boot-with-buildroot-on-qemu" class="bare">https://stackoverflow.com/questions/58028789/how-to-boot-linux-aarch64-with-u-boot-with-buildroot-on-qemu</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="emulators"><a class="anchor" href="#emulators"></a><a class="link" href="#emulators">22. Emulators</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Emulator" class="bare">https://en.wikipedia.org/wiki/Emulator</a></p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#qemu">QEMU</a></p>
</li>
<li>
<p><a href="#gem5">gem5</a></p>
</li>
<li>
<p><a href="#gensim">Gensim</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="qemu"><a class="anchor" href="#qemu"></a><a class="link" href="#qemu">23. QEMU</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="introduction-to-qemu"><a class="anchor" href="#introduction-to-qemu"></a><a class="link" href="#introduction-to-qemu">23.1. Introduction to QEMU</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/QEMU">QEMU</a> is a system simulator: it simulates a CPU and devices such as interrupt handlers, timers, UART, screen, keyboard, etc.</p>
</div>
<div class="paragraph">
<p>If you are familiar with <a href="https://en.wikipedia.org/wiki/VirtualBox">VirtualBox</a>, then QEMU then basically does the same thing: it opens a "window" inside your desktop that can run an operating system inside your operating system.</p>
</div>
<div class="paragraph">
<p>Also both can use very similar techniques: either <a href="#binary-translation">Binary translation</a> or <a href="#kvm">KVM</a>. VirtualBox' binary translator is / was based on QEMU&#8217;s it seems: <a href="https://en.wikipedia.org/wiki/VirtualBox#Software-based_virtualization" class="bare">https://en.wikipedia.org/wiki/VirtualBox#Software-based_virtualization</a></p>
</div>
<div class="paragraph">
<p>The huge advantage of QEMU over VirtualBox is that is supports cross arch simulation, e.g. simulate an ARM guest on an x86 host.</p>
</div>
<div class="paragraph">
<p>QEMU is likely the leading cross arch system simulator as of 2018. It is even the default <a href="#android">Android</a> simulator that developers get with Android Studio 3 to develop apps without real hardware.</p>
</div>
<div class="paragraph">
<p>Another advantage of QEMU over virtual box is that it doesn&#8217;t have Oracle' hands all all over it, more like RedHat + ARM.</p>
</div>
<div class="paragraph">
<p>Another advantage of QEMU is that is has no nice configuration GUI. Because who needs GUIs when you have 50 million semi-documented CLI options? Android Studio adds a custom GUI configuration tool on top of it.</p>
</div>
<div class="paragraph">
<p>QEMU is also supported by Buildroot in-tree, see e.g.: <a href="https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_aarch64_virt_defconfig" class="bare">https://github.com/buildroot/buildroot/blob/2018.05/configs/qemu_aarch64_virt_defconfig</a> We however just build our own manually with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-qemu">build-qemu</a>, as it gives more flexibility, and building QEMU is very easy!</p>
</div>
<div class="paragraph">
<p>All of this makes QEMU the natural choice of reference system simulator for this repo.</p>
</div>
</div>
<div class="sect2">
<h3 id="binary-translation"><a class="anchor" href="#binary-translation"></a><a class="link" href="#binary-translation">23.2. Binary translation</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Binary_translation" class="bare">https://en.wikipedia.org/wiki/Binary_translation</a></p>
</div>
<div class="paragraph">
<p>Used by <a href="#qemu">QEMU</a> and <a href="#gensim">Gensim</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="disk-persistency"><a class="anchor" href="#disk-persistency"></a><a class="link" href="#disk-persistency">23.3. Disk persistency</a></h3>
<div class="paragraph">
<p>We disable disk persistency for both QEMU and gem5 by default, to prevent the emulator from putting the image in an unknown state.</p>
</div>
<div class="paragraph">
<p>For QEMU, this is done by passing the <code>snapshot</code> option to <code>-drive</code>, and for gem5 it is the default behaviour.</p>
</div>
<div class="paragraph">
<p>If you hack up our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> script to remove that option, then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'date &gt;f;poweroff'</pre>
</div>
</div>
<div class="paragraph">
<p>followed by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'cat f'</pre>
</div>
</div>
<div class="paragraph">
<p>gives the date, because <code>poweroff</code> without <code>-n</code> syncs before shutdown.</p>
</div>
<div class="paragraph">
<p>The <code>sync</code> command also saves the disk:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sync</pre>
</div>
</div>
<div class="paragraph">
<p>When you do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot</pre>
</div>
</div>
<div class="paragraph">
<p>the disk image gets overwritten by a fresh filesystem and you lose all changes.</p>
</div>
<div class="paragraph">
<p>Remember that if you forcibly turn QEMU off without <code>sync</code> or <code>poweroff</code> from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved.</p>
</div>
<div class="paragraph">
<p>Persistency is also turned off when booting from <a href="#initrd">initrd</a> with a CPIO instead of with a disk.</p>
</div>
<div class="paragraph">
<p>Disk persistency is useful to re-run shell commands from the history of a previous session with <code>Ctrl-R</code>, but we felt that the loss of determinism was not worth it.</p>
</div>
<div class="sect3">
<h4 id="gem5-disk-persistency"><a class="anchor" href="#gem5-disk-persistency"></a><a class="link" href="#gem5-disk-persistency">23.3.1. gem5 disk persistency</a></h4>
<div class="paragraph">
<p>TODO how to make gem5 disk writes persistent?</p>
</div>
<div class="paragraph">
<p>As of cadb92f2df916dbb47f428fd1ec4932a2e1f0f48 there are some <code>read_only</code> entries in the <a href="#gem5-config-ini">gem5 config.ini</a> under cow sections, but hacking them to true did not work:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff --git a/configs/common/FSConfig.py b/configs/common/FSConfig.py
index 17498c42b..76b8b351d 100644
--- a/configs/common/FSConfig.py
+++ b/configs/common/FSConfig.py
@@ -60,7 +60,7 @@ os_types = { 'alpha' : [ 'linux' ],
            }

 class CowIdeDisk(IdeDisk):
-    image = CowDiskImage(child=RawDiskImage(read_only=True),
+    image = CowDiskImage(child=RawDiskImage(read_only=False),
                          read_only=False)

     def childImage(self, ci):</pre>
</div>
</div>
<div class="paragraph">
<p>The directory of interest is <code>src/dev/storage</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-qcow2"><a class="anchor" href="#gem5-qcow2"></a><a class="link" href="#gem5-qcow2">23.4. gem5 qcow2</a></h3>
<div class="paragraph">
<p>qcow2 does not appear supported, there are not hits in the source tree, and there is a mention on Nate&#8217;s 2009 wishlist: <a href="http://gem5.org/Nate%27s_Wish_List" class="bare">http://gem5.org/Nate%27s_Wish_List</a></p>
</div>
<div class="paragraph">
<p>This would be good to allow storing smaller sparse ext2 images locally on disk.</p>
</div>
</div>
<div class="sect2">
<h3 id="snapshot"><a class="anchor" href="#snapshot"></a><a class="link" href="#snapshot">23.5. Snapshot</a></h3>
<div class="paragraph">
<p>QEMU allows us to take snapshots at any time through the monitor.</p>
</div>
<div class="paragraph">
<p>You can then restore CPU, memory and disk state back at any time.</p>
</div>
<div class="paragraph">
<p>qcow2 filesystems must be used for that to work.</p>
</div>
<div class="paragraph">
<p>To test it out, login into the VM with and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'umount /mnt/9p/*;./count.sh'</pre>
</div>
</div>
<div class="paragraph">
<p>On another shell, take a snapshot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor savevm my_snap_id</pre>
</div>
</div>
<div class="paragraph">
<p>The counting continues.</p>
</div>
<div class="paragraph">
<p>Restore the snapshot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor loadvm my_snap_id</pre>
</div>
</div>
<div class="paragraph">
<p>and the counting goes back to where we saved. This shows that CPU and memory states were reverted.</p>
</div>
<div class="paragraph">
<p>The <code>umount</code> is needed because snapshotting conflicts with <a href="#9p">9P</a>, which we felt is a more valuable default. If you forget to unmount, the following error appears on the QEMU monitor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Migration is disabled when VirtFS export path '/linux-kernel-module-cheat/out/x86_64/buildroot/build' is mounted in the guest using mount_tag 'host_out'</pre>
</div>
</div>
<div class="paragraph">
<p>We can also verify that the disk state is also reversed. Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 0 &gt;f</pre>
</div>
</div>
<div class="paragraph">
<p>Monitor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor savevm my_snap_id</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt;f</pre>
</div>
</div>
<div class="paragraph">
<p>Monitor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor loadvm my_snap_id</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat f</pre>
</div>
</div>
<div class="paragraph">
<p>And the output is <code>0</code>.</p>
</div>
<div class="paragraph">
<p>Our setup does not allow for snapshotting while using <a href="#initrd">initrd</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371" class="bare">https://stackoverflow.com/questions/40227651/does-qemu-emulator-have-checkpoint-function/48724371#48724371</a></p>
</div>
<div class="sect3">
<h4 id="snapshot-internals"><a class="anchor" href="#snapshot-internals"></a><a class="link" href="#snapshot-internals">23.5.1. Snapshot internals</a></h4>
<div class="paragraph">
<p>Snapshots are stored inside the <code>.qcow2</code> images themselves.</p>
</div>
<div class="paragraph">
<p>They can be observed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>"$(./getvar buildroot_host_dir)/bin/qemu-img" info "$(./getvar qcow2_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>which after <code>savevm my_snap_id</code> and <code>savevm asdf</code> contains an output of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>image: out/x86_64/buildroot/images/rootfs.ext2.qcow2
file format: qcow2
virtual size: 512M (536870912 bytes)
disk size: 180M
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         my_snap_id              47M 2018-04-27 21:17:50   00:00:15.251
2         asdf                    47M 2018-04-27 21:20:39   00:00:18.583
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false</pre>
</div>
</div>
<div class="paragraph">
<p>As a consequence:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>it is possible to restore snapshots across boots, since they stay on the same image the entire time</p>
</li>
<li>
<p>it is not possible to use snapshots with <a href="#initrd">initrd</a> in our setup, since we don&#8217;t pass <code>-drive</code> at all when initrd is enabled</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="device-models"><a class="anchor" href="#device-models"></a><a class="link" href="#device-models">23.6. Device models</a></h3>
<div class="paragraph">
<p>This section documents:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>how to interact with peripheral hardware device models through device drivers</p>
</li>
<li>
<p>how to write your own hardware device models for our emulators, see also: <a href="https://stackoverflow.com/questions/28315265/how-to-add-a-new-device-in-qemu-source-code" class="bare">https://stackoverflow.com/questions/28315265/how-to-add-a-new-device-in-qemu-source-code</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For the more complex interfaces, we focus on simplified educational devices, either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>present in the QEMU upstream:</p>
<div class="ulist">
<ul>
<li>
<p><a href="#qemu-edu">QEMU edu PCI device</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="pci"><a class="anchor" href="#pci"></a><a class="link" href="#pci">23.6.1. PCI</a></h4>
<div class="paragraph">
<p>Only tested in x86.</p>
</div>
<div class="sect4">
<h5 id="qemu-edu"><a class="anchor" href="#qemu-edu"></a><a class="link" href="#qemu-edu">23.6.1.1. QEMU edu PCI device</a></h5>
<div class="paragraph">
<p>Small upstream educational PCI device:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu_edu.sh</pre>
</div>
</div>
<div class="paragraph">
<p>This tests a lot of features of the edu device, to understand the results, compare the inputs with the documentation of the hardware: <a href="https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt" class="bare">https://github.com/qemu/qemu/blob/v2.12.0/docs/specs/edu.txt</a></p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>kernel module: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/qemu_edu.c">kernel_modules/qemu_edu.c</a></p>
</li>
<li>
<p>QEMU device: <a href="https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c" class="bare">https://github.com/qemu/qemu/blob/v2.12.0/hw/misc/edu.c</a></p>
</li>
<li>
<p>test script: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/qemu_edu.sh">rootfs_overlay/lkmc/qemu_edu.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Works because we add to our default QEMU CLI:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-device edu</pre>
</div>
</div>
<div class="paragraph">
<p>This example uses:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the QEMU <code>edu</code> educational device, which is a minimal educational in-tree PCI example</p>
</li>
<li>
<p>the <code>pci.ko</code> kernel module, which exercises the <code>edu</code> hardware.</p>
<div class="paragraph">
<p>I&#8217;ve contacted the awesome original author author of <code>edu</code> <a href="https://github.com/jirislaby">Jiri Slaby</a>, and he told there is no official kernel module example because this was created for a kernel module university course that he gives, and he didn&#8217;t want to give away answers. <a href="https://github.com/cirosantilli/how-to-teach-efficiently">I don&#8217;t agree with that philosophy</a>, so students, cheat away with this repo and go make startups instead.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO exercise DMA on the kernel module. The <code>edu</code> hardware model has that feature:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/17913679/how-to-instantiate-and-use-a-dma-driver-linux-module" class="bare">https://stackoverflow.com/questions/17913679/how-to-instantiate-and-use-a-dma-driver-linux-module</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/32592734/are-there-any-dma-driver-example-pcie-and-fpga/44716747#44716747" class="bare">https://stackoverflow.com/questions/32592734/are-there-any-dma-driver-example-pcie-and-fpga/44716747#44716747</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/62831327/add-memory-device-to-qemu" class="bare">https://stackoverflow.com/questions/62831327/add-memory-device-to-qemu</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/64539528/qemu-pci-dma-read-and-pci-dma-write-does-not-work" class="bare">https://stackoverflow.com/questions/64539528/qemu-pci-dma-read-and-pci-dma-write-does-not-work</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/64842929/general-protection-error-while-tring-to-perform-ioctl" class="bare">https://stackoverflow.com/questions/64842929/general-protection-error-while-tring-to-perform-ioctl</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="manipulate-pci-registers-directly"><a class="anchor" href="#manipulate-pci-registers-directly"></a><a class="link" href="#manipulate-pci-registers-directly">23.6.1.2. Manipulate PCI registers directly</a></h5>
<div class="paragraph">
<p>In this section we will try to interact with PCI devices directly from userland without kernel modules.</p>
</div>
<div class="paragraph">
<p>First identify the PCI device with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lspci</pre>
</div>
</div>
<div class="paragraph">
<p>In our case for example, we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>00:06.0 Unclassified device [00ff]: Device 1234:11e8 (rev 10)
00:07.0 Unclassified device [00ff]: Device 1234:11e9</pre>
</div>
</div>
<div class="paragraph">
<p>which we identify as being <a href="#qemu-edu">QEMU edu PCI device</a> by the magic number: <code>1234:11e8</code>.</p>
</div>
<div class="paragraph">
<p>Alternatively, we can also do use the QEMU monitor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor info qtree</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      dev: edu, id ""
        addr = 06.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        class Class 00ff, addr 00:06.0, pci id 1234:11e8 (sub 1af4:1100)
        bar 0: mem at 0xfea00000 [0xfeafffff]</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="https://serverfault.com/questions/587189/list-all-devices-emulated-for-a-vm/913622#913622" class="bare">https://serverfault.com/questions/587189/list-all-devices-emulated-for-a-vm/913622#913622</a></p>
</div>
<div class="paragraph">
<p>Read the configuration registers as binary:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>hexdump /sys/bus/pci/devices/0000:00:06.0/config</pre>
</div>
</div>
<div class="paragraph">
<p>Get nice human readable names and offsets of the registers and some enums:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>setpci --dumpregs</pre>
</div>
</div>
<div class="paragraph">
<p>Get the values of a given config register from its human readable name, either with either bus or device id:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>setpci -s 0000:00:06.0 BASE_ADDRESS_0
setpci -d 1234:11e8 BASE_ADDRESS_0</pre>
</div>
</div>
<div class="paragraph">
<p>Note however that <code>BASE_ADDRESS_0</code> also appears when you do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lspci -v</pre>
</div>
</div>
<div class="paragraph">
<p>as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Memory at feb54000</pre>
</div>
</div>
<div class="paragraph">
<p>Then you can try messing with that address with <a href="#dev-mem">/dev/mem</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>devmem 0xfeb54000 w 0x12345678</pre>
</div>
</div>
<div class="paragraph">
<p>which writes to the first register of the edu device.</p>
</div>
<div class="paragraph">
<p>The device then fires an interrupt at irq 11, which is unhandled, which leads the kernel to say you are a bad person:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;3&gt;[ 1065.567742] irq 11: nobody cared (try booting with the "irqpoll" option)</pre>
</div>
</div>
<div class="paragraph">
<p>followed by a trace.</p>
</div>
<div class="paragraph">
<p>Next, also try using our <a href="#irq-ko">irq.ko</a> IRQ monitoring module before triggering the interrupt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>insmod irq.ko
devmem 0xfeb54000 w 0x12345678</pre>
</div>
</div>
<div class="paragraph">
<p>Our kernel module handles the interrupt, but does not acknowledge it like our proper edu kernel module, and so it keeps firing, which leads to infinitely many messages being printed:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>handler irq = 11 dev = 251</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="pciutils"><a class="anchor" href="#pciutils"></a><a class="link" href="#pciutils">23.6.1.3. pciutils</a></h5>
<div class="paragraph">
<p>There are two versions of <code>setpci</code> and <code>lspci</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a simple one from BusyBox</p>
</li>
<li>
<p>a more complete one from <a href="https://github.com/pciutils/pciutils">pciutils</a> which Buildroot has a package for, and is the default on Ubuntu 18.04 host. This is the one we enable by default.</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="introduction-to-pci"><a class="anchor" href="#introduction-to-pci"></a><a class="link" href="#introduction-to-pci">23.6.1.4. Introduction to PCI</a></h5>
<div class="paragraph">
<p>The PCI standard is non-free, obviously like everything in low level: <a href="https://pcisig.com/specifications" class="bare">https://pcisig.com/specifications</a> but Google gives several illegal PDF hits :-)</p>
</div>
<div class="paragraph">
<p>And of course, the best documentation available is: <a href="http://wiki.osdev.org/PCI" class="bare">http://wiki.osdev.org/PCI</a></p>
</div>
<div class="paragraph">
<p>Like every other hardware, we could interact with PCI on x86 using only IO instructions and memory operations.</p>
</div>
<div class="paragraph">
<p>But PCI is a complex communication protocol that the Linux kernel implements beautifully for us, so let&#8217;s use the kernel API.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>edu device source and spec in QEMU tree:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/qemu/qemu/blob/v2.7.0/hw/misc/edu.c" class="bare">https://github.com/qemu/qemu/blob/v2.7.0/hw/misc/edu.c</a></p>
</li>
<li>
<p><a href="https://github.com/qemu/qemu/blob/v2.7.0/docs/specs/edu.txt" class="bare">https://github.com/qemu/qemu/blob/v2.7.0/docs/specs/edu.txt</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="http://www.zarb.org/~trem/kernel/pci/pci-driver.c" class="bare">http://www.zarb.org/~trem/kernel/pci/pci-driver.c</a> inb outb runnable example (no device)</p>
</li>
<li>
<p>LDD3 PCI chapter</p>
</li>
<li>
<p>another QEMU device + module, but using a custom QEMU device:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/levex/kernel-qemu-pci/blob/31fc9355161b87cea8946b49857447ddd34c7aa6/module/levpci.c" class="bare">https://github.com/levex/kernel-qemu-pci/blob/31fc9355161b87cea8946b49857447ddd34c7aa6/module/levpci.c</a></p>
</li>
<li>
<p><a href="https://github.com/levex/kernel-qemu-pci/blob/31fc9355161b87cea8946b49857447ddd34c7aa6/qemu/hw/char/lev-pci.c" class="bare">https://github.com/levex/kernel-qemu-pci/blob/31fc9355161b87cea8946b49857447ddd34c7aa6/qemu/hw/char/lev-pci.c</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://is.muni.cz/el/1433/podzim2016/PB173/um/65218991/" class="bare">https://is.muni.cz/el/1433/podzim2016/PB173/um/65218991/</a> course given by the creator of the edu device. In Czech, and only describes API</p>
</li>
<li>
<p><a href="http://nairobi-embedded.org/linux_pci_device_driver.html" class="bare">http://nairobi-embedded.org/linux_pci_device_driver.html</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="pci-bfd"><a class="anchor" href="#pci-bfd"></a><a class="link" href="#pci-bfd">23.6.1.5. PCI BFD</a></h5>
<div class="paragraph">
<p><code>lspci -k</code> shows something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>00:04.0 Class 00ff: 1234:11e8 lkmc_pci</pre>
</div>
</div>
<div class="paragraph">
<p>Meaning of the first numbers:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;8:bus&gt;:&lt;5:device&gt;.&lt;3:function&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>Often abbreviated to BDF.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>bus: groups PCI slots</p>
</li>
<li>
<p>device: maps to one slot</p>
</li>
<li>
<p>function: <a href="https://stackoverflow.com/questions/19223394/what-is-the-function-number-in-pci/44735372#44735372" class="bare">https://stackoverflow.com/questions/19223394/what-is-the-function-number-in-pci/44735372#44735372</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Sometimes a fourth number is also added, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0000:00:04.0</pre>
</div>
</div>
<div class="paragraph">
<p>TODO is that the domain?</p>
</div>
<div class="paragraph">
<p>Class: pure magic: <a href="https://www-s.acm.illinois.edu/sigops/2007/roll_your_own/7.c.1.html" class="bare">https://www-s.acm.illinois.edu/sigops/2007/roll_your_own/7.c.1.html</a> TODO: does it have any side effects? Set in the edu device at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>k-&gt;class_id = PCI_CLASS_OTHERS</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="pci-bar"><a class="anchor" href="#pci-bar"></a><a class="link" href="#pci-bar">23.6.1.6. PCI BAR</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/30190050/what-is-base-address-register-bar-in-pcie/44716618#44716618" class="bare">https://stackoverflow.com/questions/30190050/what-is-base-address-register-bar-in-pcie/44716618#44716618</a></p>
</div>
<div class="paragraph">
<p>Each PCI device has 6 BAR IOs (base address register) as per the PCI spec.</p>
</div>
<div class="paragraph">
<p>Each BAR corresponds to an address range that can be used to communicate with the PCI.</p>
</div>
<div class="paragraph">
<p>Each BAR is of one of the two types:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>IORESOURCE_IO</code>: must be accessed with <code>inX</code> and <code>outX</code></p>
</li>
<li>
<p><code>IORESOURCE_MEM</code>: must be accessed with <code>ioreadX</code> and <code>iowriteX</code>. This is the saner method apparently, and what the edu device uses.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The length of each region is defined by the hardware, and communicated to software via the configuration registers.</p>
</div>
<div class="paragraph">
<p>The Linux kernel automatically parses the 64 bytes of standardized configuration registers for us.</p>
</div>
<div class="paragraph">
<p>QEMU devices register those regions with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>memory_region_init_io(&amp;edu-&gt;mmio, OBJECT(edu), &amp;edu_mmio_ops, edu,
                "edu-mmio", 1 &lt;&lt; 20);
pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &amp;edu-&gt;mmio);</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gpio"><a class="anchor" href="#gpio"></a><a class="link" href="#gpio">23.6.2. GPIO</a></h4>
<div class="paragraph">
<p>TODO: broken. Was working before we moved <code>arm</code> from <code>-M versatilepb</code> to <code>-M virt</code> around af210a76711b7fa4554dcc2abd0ddacfc810dfd4. Either make it work on <code>-M virt</code> if that is possible, or document precisely how to make it work with <code>versatilepb</code>, or hopefully <code>vexpress</code> which is newer.</p>
</div>
<div class="paragraph">
<p>QEMU does not have a very nice mechanism to observe GPIO activity: <a href="https://raspberrypi.stackexchange.com/questions/56373/is-it-possible-to-get-the-state-of-the-leds-and-gpios-in-a-qemu-emulation-like-t/69267#69267" class="bare">https://raspberrypi.stackexchange.com/questions/56373/is-it-possible-to-get-the-state-of-the-leds-and-gpios-in-a-qemu-emulation-like-t/69267#69267</a></p>
</div>
<div class="paragraph">
<p>The best you can do is to hack our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build">build</a> script to add:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>HOST_QEMU_OPTS='--extra-cflags=-DDEBUG_PL061=1'</pre>
</div>
</div>
<div class="paragraph">
<p>where <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0190b/index.html">PL061</a> is the dominating ARM Holdings hardware that handles GPIO.</p>
</div>
<div class="paragraph">
<p>Then compile with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch arm --config-fragment buildroot_config/gpio
./build-linux --config-fragment linux_config/gpio</pre>
</div>
</div>
<div class="paragraph">
<p>then test it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gpio.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/gpio.sh">rootfs_overlay/lkmc/gpio.sh</a></p>
</div>
<div class="paragraph">
<p>Buildroot&#8217;s Linux tools package provides some GPIO CLI tools: <code>lsgpio</code>, <code>gpio-event-mon</code>, <code>gpio-hammer</code>, TODO document them here.</p>
</div>
</div>
<div class="sect3">
<h4 id="leds"><a class="anchor" href="#leds"></a><a class="link" href="#leds">23.6.3. LEDs</a></h4>
<div class="paragraph">
<p>TODO: broken when <code>arm</code> moved to <code>-M virt</code>, same as <a href="#gpio">GPIO</a>.</p>
</div>
<div class="paragraph">
<p>Hack QEMU&#8217;s <code>hw/misc/arm_sysctl.c</code> with a printf:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static void arm_sysctl_write(void *opaque, hwaddr offset,
                            uint64_t val, unsigned size)
{
    arm_sysctl_state *s = (arm_sysctl_state *)opaque;

    switch (offset) {
    case 0x08: /* LED */
        printf("LED val = %llx\n", (unsigned long long)val);</pre>
</div>
</div>
<div class="paragraph">
<p>and then rebuild with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-qemu --arch arm
./build-linux --arch arm --config-fragment linux_config/leds</pre>
</div>
</div>
<div class="paragraph">
<p>But beware that one of the LEDs has a heartbeat trigger by default (specified on dts), so it will produce a lot of output.</p>
</div>
<div class="paragraph">
<p>And then activate it with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd /sys/class/leds/versatile:0
cat max_brightness
echo 255 &gt;brightness</pre>
</div>
</div>
<div class="paragraph">
<p>Relevant QEMU files:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>hw/arm/versatilepb.c</code></p>
</li>
<li>
<p><code>hw/misc/arm_sysctl.c</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Relevant kernel files:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>arch/arm/boot/dts/versatile-pb.dts</code></p>
</li>
<li>
<p><code>drivers/leds/led-class.c</code></p>
</li>
<li>
<p><code>drivers/leds/leds-sysctl.c</code></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gem5-educational-hardware-models"><a class="anchor" href="#gem5-educational-hardware-models"></a><a class="link" href="#gem5-educational-hardware-models">23.6.4. gem5 educational hardware models</a></h4>
<div class="paragraph">
<p>TODO get some working!</p>
</div>
<div class="paragraph">
<p><a href="http://gedare-csphd.blogspot.co.uk/2013/02/adding-simple-io-device-to-gem5.html" class="bare">http://gedare-csphd.blogspot.co.uk/2013/02/adding-simple-io-device-to-gem5.html</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qemu-monitor"><a class="anchor" href="#qemu-monitor"></a><a class="link" href="#qemu-monitor">23.7. QEMU monitor</a></h3>
<div class="paragraph">
<p>The QEMU monitor is a magic terminal that allows you to send text commands to the QEMU VM itself: <a href="https://en.wikibooks.org/wiki/QEMU/Monitor" class="bare">https://en.wikibooks.org/wiki/QEMU/Monitor</a></p>
</div>
<div class="paragraph">
<p>While QEMU is running, on another terminal, run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor</pre>
</div>
</div>
<div class="paragraph">
<p>or send one command such as <code>info qtree</code> and quit the monitor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-monitor info qtree</pre>
</div>
</div>
<div class="paragraph">
<p>or equivalently:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 'info qtree' | ./qemu-monitor</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/qemu-monitor">qemu-monitor</a></p>
</div>
<div class="paragraph">
<p><code>qemu-monitor</code> uses the <code>-monitor</code> QEMU command line option, which makes the monitor listen from a socket.</p>
</div>
<div class="paragraph">
<p>Alternatively, we can also enter the QEMU monitor from inside <code>-nographics</code> <a href="#qemu-text-mode">QEMU text mode</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-A C</pre>
</div>
</div>
<div class="paragraph">
<p>and go back to the terminal with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-A C</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/14165158/how-to-switch-to-qemu-monitor-console-when-running-with-curses" class="bare">https://stackoverflow.com/questions/14165158/how-to-switch-to-qemu-monitor-console-when-running-with-curses</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/488263/how-to-switch-to-the-qemu-control-panel-with-nographics" class="bare">https://superuser.com/questions/488263/how-to-switch-to-the-qemu-control-panel-with-nographics</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>When in graphic mode, we can do it from the GUI:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-Alt ?</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>?</code> is a digit <code>1</code>, or <code>2</code>, or, <code>3</code>, etc. depending on what else is available on the GUI: serial, parallel and frame buffer.</p>
</div>
<div class="paragraph">
<p>Finally, we can also access QEMU monitor commands directly from <a href="#gdb">GDB step debug</a> with the <code>monitor</code> command:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb</pre>
</div>
</div>
<div class="paragraph">
<p>then inside that shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>monitor info qtree</pre>
</div>
</div>
<div class="paragraph">
<p>This way you can use both QEMU monitor and GDB commands to inspect the guest from inside a single shell! Pretty awesome.</p>
</div>
<div class="paragraph">
<p>In general, <code>./qemu-monitor</code> is the best option, as it:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>works on both modes</p>
</li>
<li>
<p>allows to use the host Bash history to re-run one off commands</p>
</li>
<li>
<p>allows you to search the output of commands on your host shell even when in graphic mode</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Getting everything to work required careful choice of QEMU command line options:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/49716931/how-to-run-qemu-with-nographic-and-monitor-but-still-be-able-to-send-ctrlc-to/49751144#49751144" class="bare">https://stackoverflow.com/questions/49716931/how-to-run-qemu-with-nographic-and-monitor-but-still-be-able-to-send-ctrlc-to/49751144#49751144</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/167165/how-to-pass-ctrl-c-to-the-guest-when-running-qemu-with-nographic/436321#436321" class="bare">https://unix.stackexchange.com/questions/167165/how-to-pass-ctrl-c-to-the-guest-when-running-qemu-with-nographic/436321#436321</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="qemu-monitor-from-guest"><a class="anchor" href="#qemu-monitor-from-guest"></a><a class="link" href="#qemu-monitor-from-guest">23.7.1. QEMU monitor from guest</a></h4>
<div class="paragraph">
<p>Peter Maydell said potentially not possible nicely as of August 2018: <a href="https://stackoverflow.com/questions/51747744/how-to-run-a-qemu-monitor-command-from-inside-the-guest/51764110#51764110" class="bare">https://stackoverflow.com/questions/51747744/how-to-run-a-qemu-monitor-command-from-inside-the-guest/51764110#51764110</a></p>
</div>
<div class="paragraph">
<p>It is also worth looking into the QEMU Guest Agent tool <code>qemu-gq</code> that can be enabled with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_QEMU=y'</pre>
</div>
</div>
<div class="paragraph">
<p>See also: <a href="https://superuser.com/questions/930588/how-to-pass-commands-noninteractively-to-running-qemu-from-the-guest-qmp-via-te" class="bare">https://superuser.com/questions/930588/how-to-pass-commands-noninteractively-to-running-qemu-from-the-guest-qmp-via-te</a></p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-monitor-from-gdb"><a class="anchor" href="#qemu-monitor-from-gdb"></a><a class="link" href="#qemu-monitor-from-gdb">23.7.2. QEMU monitor from GDB</a></h4>
<div class="paragraph">
<p>When doing <a href="#gdb">GDB step debug</a> it is possible to send QEMU monitor commands through the GDB <code>monitor</code> command, which saves you the trouble of opening yet another shell.</p>
</div>
<div class="paragraph">
<p>Try for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>monitor help
monitor info qtree</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="debug-the-emulator"><a class="anchor" href="#debug-the-emulator"></a><a class="link" href="#debug-the-emulator">23.8. Debug the emulator</a></h3>
<div class="paragraph">
<p>When you start hacking QEMU or gem5, it is useful to see what is going on inside the emulator themselves.</p>
</div>
<div class="paragraph">
<p>This is of course trivial since they are just regular userland programs on the host, but we make it a bit easier with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm</pre>
</div>
</div>
<div class="paragraph">
<p>Or for a faster development loop you can pass <code>-ex</code> command as a semicolon separated list:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm-ex 'break qemu_add_opts;run'</pre>
</div>
</div>
<div class="paragraph">
<p>which is equivalent to the more verbose:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm-args '-ex "break qemu_add_opts" -ex "run"'</pre>
</div>
</div>
<div class="paragraph">
<p>if you ever want need anything besides -ex.</p>
</div>
<div class="paragraph">
<p>Or if things get really involved and you want a debug script:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf 'break qemu_add_opts
run
' &gt; data/vm.gdb
./run --debug-vm-file data/vm.gdb</pre>
</div>
</div>
<div class="paragraph">
<p>Our default emulator builds are optimized with <code>gcc -O2 -g</code>. To use <code>-O0</code> instead, build and run with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-qemu --qemu-build-type debug --verbose
./run --debug-vm
./build-gem5 --gem5-build-type debug --verbose
./run --debug-vm --emulator-gem5</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>--verbose</code> is optional, but shows clearly each GCC build command so that you can confirm what <code>--*-build-type</code> is doing.</p>
</div>
<div class="paragraph">
<p>The build outputs are automatically stored in a different directories for optimized and debug builds, which prevents <code>debug</code> files from overwriting <code>opt</code> ones. Therefore, <code>--gem5-build-id</code> is not required.</p>
</div>
<div class="paragraph">
<p>The price to pay for debuggability is high however: a Linux kernel boot was about 3x slower in QEMU and 14 times slower in gem5 debug compared to opt, see benchmarks at: <a href="#benchmark-linux-kernel-boot">Section 35.2.1, &#8220;Benchmark Linux kernel boot&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Similar slowdowns can be observed at: <a href="#benchmark-emulators-on-userland-executables">Section 35.2.2, &#8220;Benchmark emulators on userland executables&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>When in <a href="#qemu-text-mode">QEMU text mode</a>, using <code>--debug-vm</code> makes Ctrl-C not get passed to the QEMU guest anymore: it is instead captured by GDB itself, so allow breaking. So e.g. you won&#8217;t be able to easily quit from a guest program like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sleep 10</pre>
</div>
</div>
<div class="paragraph">
<p>In graphic mode, make sure that you never click inside the QEMU graphic while debugging, otherwise you mouse gets captured forever, and the only solution I can find is to go to a TTY with <code>Ctrl-Alt-F1</code> and <code>kill</code> QEMU.</p>
</div>
<div class="paragraph">
<p>You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.</p>
</div>
<div class="sect3">
<h4 id="reverse-debug-the-emulator"><a class="anchor" href="#reverse-debug-the-emulator"></a><a class="link" href="#reverse-debug-the-emulator">23.8.1. Reverse debug the emulator</a></h4>
<div class="paragraph">
<p>While step debugging any complex program, you always end up feeling the need to step in reverse to reach the last call to some function that was called before the failure point, in order to trace back the problem to the actual bug source.</p>
</div>
<div class="paragraph">
<p>While GDB "has" this feature, it is just too broken to be usable, and so we expose the amazing Mozilla RR tool conveniently in this repo: <a href="https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242" class="bare">https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242</a></p>
</div>
<div class="paragraph">
<p>Before the first usage setup rr with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 'kernel.perf_event_paranoid=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p</pre>
</div>
</div>
<div class="paragraph">
<p>Then use it with your content of interest, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm-rr --userland userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>This will:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>first run the program once until completion or crash</p>
</li>
<li>
<p>then restart the program at the very first instruction at <code>_start</code> and leave you in a GDB shell</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>From there, run the program until your point of interest, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>break qemu_add_opts
continue</pre>
</div>
</div>
<div class="paragraph">
<p>and you can now reliably use reverse debugging commands such as <code>reverse-continue</code>, <code>reverse-finish</code> and <code>reverse-next</code>!</p>
</div>
<div class="paragraph">
<p>To restart debugging again after quitting <code>rr</code>, simply run on your host terminal:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rr replay</pre>
</div>
</div>
<div class="paragraph">
<p>The use case of <code>rr</code> is often to go to the final crash and then walk back from there, so you often want to automate running until the end after record with <code>--debug-vm-args</code> as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --debug-vm-args='-ex continue' --debug-vm-rr --userland userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>Programs often tend to blow up in very low frames that use values passed in from higher frames. In those cases, remember that just like with forward debugging, you can&#8217;t just go:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>up
up
up
reverse-next</pre>
</div>
</div>
<div class="paragraph">
<p>but rather, you must:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>reverse-finish
reverse-finish
reverse-finish
reverse-next</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="debug-gem5-python-scripts"><a class="anchor" href="#debug-gem5-python-scripts"></a><a class="link" href="#debug-gem5-python-scripts">23.8.2. Debug gem5 Python scripts</a></h4>
<div class="paragraph">
<p>Start pdb at the first instruction:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --gem5-exe-args='--pdb' --terminal</pre>
</div>
</div>
<div class="paragraph">
<p>Requires <code>--terminal</code> as we must be on foreground.</p>
</div>
<div class="paragraph">
<p>Alternatively, you can add to the point of the code where you want to break the usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>import ipdb; ipdb.set_trace()</pre>
</div>
</div>
<div class="paragraph">
<p>and then run with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --terminal</pre>
</div>
</div>
<div class="paragraph">
<p>TODO test PyCharm: <a href="https://stackoverflow.com/questions/51982735/writing-gem5-configuration-scripts-with-pycharm" class="bare">https://stackoverflow.com/questions/51982735/writing-gem5-configuration-scripts-with-pycharm</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="tracing"><a class="anchor" href="#tracing"></a><a class="link" href="#tracing">23.9. Tracing</a></h3>
<div class="paragraph">
<p>QEMU can log several different events.</p>
</div>
<div class="paragraph">
<p>The most interesting are events which show instructions that QEMU ran, for which we have a helper:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./trace-boot --arch x86_64</pre>
</div>
</div>
<div class="paragraph">
<p>Under the hood, this uses QEMU&#8217;s <code>-trace</code> option.</p>
</div>
<div class="paragraph">
<p>You can then inspect the address of each instruction run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar --arch x86_64 run_dir)/trace.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output excerpt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>exec_tb 0.000 pid=10692 tb=0x7fb4f8000040 pc=0xfffffff0
exec_tb 35.391 pid=10692 tb=0x7fb4f8000180 pc=0xfe05b
exec_tb 21.047 pid=10692 tb=0x7fb4f8000340 pc=0xfe066
exec_tb 12.197 pid=10692 tb=0x7fb4f8000480 pc=0xfe06a</pre>
</div>
</div>
<div class="paragraph">
<p>Get the list of available trace events:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --trace help</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: any way to show the actualy disassembled instruction executed directly from there? Possible with <a href="#qemu-d-tracing">QEMU -d tracing</a>.</p>
</div>
<div class="paragraph">
<p>Enable other specific trace events:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --trace trace1,trace2
./qemu-trace2txt -a "$arch"
less "$(./getvar -a "$arch" run_dir)/trace.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>This functionality relies on the following setup:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>./configure --enable-trace-backends=simple</code>. This logs in a binary format to the trace file.</p>
<div class="paragraph">
<p>It makes 3x execution faster than the default trace backend which logs human readable data to stdout.</p>
</div>
<div class="paragraph">
<p>Logging with the default backend <code>log</code> greatly slows down the CPU, and in particular leads to this boot message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>All QSes seen, last rcu_sched kthread activity 5252 (4294901421-4294896169), jiffies_till_next_fqs=1, root -&gt;qsmask 0x0
swapper/0       R  running task        0     1      0 0x00000008
 ffff880007c03ef8 ffffffff8107aa5d ffff880007c16b40 ffffffff81a3b100
 ffff880007c03f60 ffffffff810a41d1 0000000000000000 0000000007c03f20
 fffffffffffffedc 0000000000000004 fffffffffffffedc ffffffff00000000
Call Trace:
 &lt;IRQ&gt;  [&lt;ffffffff8107aa5d&gt;] sched_show_task+0xcd/0x130
 [&lt;ffffffff810a41d1&gt;] rcu_check_callbacks+0x871/0x880
 [&lt;ffffffff810a799f&gt;] update_process_times+0x2f/0x60</pre>
</div>
</div>
<div class="paragraph">
<p>in which the boot appears to hang for a considerable time.</p>
</div>
</li>
<li>
<p>patch  QEMU source to remove the <code>disable</code> from <code>exec_tb</code> in the <code>trace-events</code> file. See also: <a href="https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/" class="bare">https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="qemu-d-tracing"><a class="anchor" href="#qemu-d-tracing"></a><a class="link" href="#qemu-d-tracing">23.9.1. QEMU -d tracing</a></h4>
<div class="paragraph">
<p>QEMU also has a second trace mechanism in addition to <code>-trace</code>, find out the events with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run -- -d help</pre>
</div>
</div>
<div class="paragraph">
<p>Let&#8217;s pick the one that dumps executed instructions, <code>in_asm</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval './linux/poweroff.out' -- -D out/trace.txt -d in_asm
less out/trace.txt</pre>
</div>
</div>
<div class="paragraph">
<p>Sample output excerpt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>----------------
IN:
0xfffffff0:  ea 5b e0 00 f0           ljmpw    $0xf000:$0xe05b

----------------
IN:
0x000fe05b:  2e 66 83 3e 88 61 00     cmpl     $0, %cs:0x6188
0x000fe062:  0f 85 7b f0              jne      0xd0e1</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: after <code>IN:</code>, symbol names are meant to show, which is awesome, but I don&#8217;t get any. I do see them however when running a bare metal example from: <a href="https://github.com/cirosantilli/newlib-examples/tree/900a9725947b1f375323c7da54f69e8049158881" class="bare">https://github.com/cirosantilli/newlib-examples/tree/900a9725947b1f375323c7da54f69e8049158881</a></p>
</div>
<div class="paragraph">
<p>TODO: what is the point of having two mechanisms, <code>-trace</code> and <code>-d</code>? <code>-d</code> tracing is cool because it does not require a messy recompile, and it can also show symbols.</p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-trace-register-values"><a class="anchor" href="#qemu-trace-register-values"></a><a class="link" href="#qemu-trace-register-values">23.9.2. QEMU trace register values</a></h4>
<div class="paragraph">
<p>TODO: is it possible to show the register values for each instruction?</p>
</div>
<div class="paragraph">
<p>This would include the memory values read into the registers.</p>
</div>
<div class="paragraph">
<p>Asked at: <a href="https://superuser.com/questions/1377764/how-to-trace-the-register-values-of-executed-instructions-in-qemu" class="bare">https://superuser.com/questions/1377764/how-to-trace-the-register-values-of-executed-instructions-in-qemu</a></p>
</div>
<div class="paragraph">
<p>Seems impossible due to optimizations that QEMU does:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg07479.html" class="bare">https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg07479.html</a></p>
</li>
<li>
<p><a href="https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg02856.html" class="bare">https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg02856.html</a></p>
</li>
<li>
<p><a href="https://lists.gnu.org/archive/html/qemu-devel/2012-08/msg03057.html" class="bare">https://lists.gnu.org/archive/html/qemu-devel/2012-08/msg03057.html</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>PANDA can list memory addresses, so I bet it can also decode the instructions: <a href="https://github.com/panda-re/panda/blob/883c85fa35f35e84a323ed3d464ff40030f06bd6/panda/docs/LINE_Censorship.md" class="bare">https://github.com/panda-re/panda/blob/883c85fa35f35e84a323ed3d464ff40030f06bd6/panda/docs/LINE_Censorship.md</a> I wonder why they don&#8217;t just upstream those things to QEMU&#8217;s tracing: <a href="https://github.com/panda-re/panda/issues/290" class="bare">https://github.com/panda-re/panda/issues/290</a></p>
</div>
<div class="paragraph">
<p>gem5 can do it as shown at: <a href="#gem5-tracing">Section 23.9.8, &#8220;gem5 tracing&#8221;</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-trace-memory-accesses"><a class="anchor" href="#qemu-trace-memory-accesses"></a><a class="link" href="#qemu-trace-memory-accesses">23.9.3. QEMU trace memory accesses</a></h4>
<div class="paragraph">
<p>Not possible apparently, not even with the <code>memory_region_ops_read</code> and <code>memory_region_ops_write</code> trace events, Peter comments <a href="https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg07482.html" class="bare">https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg07482.html</a></p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>No. You will miss all the fast-path memory accesses, which are
done with custom generated assembly in the TCG backend. In
general QEMU is not designed to support this kind of monitoring
of guest operations.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Related question: <a href="https://reverseengineering.stackexchange.com/questions/12260/how-to-log-all-memory-accesses-read-and-write-including-the-memory-content-in" class="bare">https://reverseengineering.stackexchange.com/questions/12260/how-to-log-all-memory-accesses-read-and-write-including-the-memory-content-in</a></p>
</div>
</div>
<div class="sect3">
<h4 id="trace-source-lines"><a class="anchor" href="#trace-source-lines"></a><a class="link" href="#trace-source-lines">23.9.4. Trace source lines</a></h4>
<div class="paragraph">
<p>We can further use Binutils' <code>addr2line</code> to get the line that corresponds to each address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./trace-boot --arch x86_64
./trace2line --arch x86_64
less "$(./getvar --arch x86_64 run_dir)/trace-lines.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>The last commands takes several seconds.</p>
</div>
<div class="paragraph">
<p>The format is as follows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>39368 _static_cpu_has arch/x86/include/asm/cpufeature.h:148</pre>
</div>
</div>
<div class="paragraph">
<p>Where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>39368</code>: number of consecutive times that a line ran. Makes the output much shorter and more meaningful</p>
</li>
<li>
<p><code>_static_cpu_has</code>: name of the function that contains the line</p>
</li>
<li>
<p><code>arch/x86/include/asm/cpufeature.h:148</code>: file and line</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This could of course all be done with GDB, but it would likely be too slow to be practical.</p>
</div>
<div class="paragraph">
<p>TODO do even more awesome offline post-mortem analysis things, such as:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>detect if we are in userspace or kernelspace. Should be a simple matter of reading the</p>
</li>
<li>
<p>read kernel data structures, and determine the current thread. Maybe we can reuse / extend the kernel&#8217;s GDB Python scripts??</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="qemu-record-and-replay"><a class="anchor" href="#qemu-record-and-replay"></a><a class="link" href="#qemu-record-and-replay">23.9.5. QEMU record and replay</a></h4>
<div class="paragraph">
<p>QEMU runs, unlike gem5, are not deterministic by default, however it does support a record and replay mechanism that allows you to replay a previous run deterministically.</p>
</div>
<div class="paragraph">
<p>This awesome feature allows you to examine a single run as many times as you would like until you understand everything:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Record a run.
./run --eval-after './linux/rand_check.out;./linux/poweroff.out;' --record
# Replay the run.
./run --eval-after './linux/rand_check.out;./linux/poweroff.out;' --replay</pre>
</div>
</div>
<div class="paragraph">
<p>A convenient shortcut to do both at once to test the feature is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-rr --eval-after './linux/rand_check.out;./linux/poweroff.out;'</pre>
</div>
</div>
<div class="paragraph">
<p>By comparing the terminal output of both runs, we can see that they are the exact same, including things which normally differ across runs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>timestamps of dmesg output</p>
</li>
<li>
<p><a href="#rand-check-out">rand_check.out</a> output</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The record and replay feature was revived around QEMU v3.0.0. In v5.2.0 it is quite usable, almost all peripherals and vCPUs are supported.</p>
</div>
<div class="paragraph">
<p>Documented at: <a href="https://github.com/qemu/qemu/blob/v5.2.0/docs/replay.txt" class="bare">https://github.com/qemu/qemu/blob/v5.2.0/docs/replay.txt</a></p>
</div>
<div class="paragraph">
<p>replay may be used with with network:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-rr --eval-after 'ifup -a;wget -S google.com;./linux/poweroff.out;'</pre>
</div>
</div>
<div class="paragraph">
<p><code>arm</code> and <code>aarch64</code> targets can also be used with rr:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./qemu-rr --arch aarch64 --eval-after './linux/rand_check.out;./linux/poweroff.out;'
./qemu-rr --arch aarch64 --eval-after 'ifup -a;wget -S google.com;./linux/poweroff.out;'</pre>
</div>
</div>
<div class="paragraph">
<p>Replay also supports <a href="#initrd">initrd</a> and no disk:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch aarch64 --initrd
./qemu-rr --arch aarch64 --eval-after './linux/rand_check.out;./linux/poweroff.out;' --initrd</pre>
</div>
</div>
<div class="sect4">
<h5 id="qemu-reverse-debugging"><a class="anchor" href="#qemu-reverse-debugging"></a><a class="link" href="#qemu-reverse-debugging">23.9.5.1. QEMU reverse debugging</a></h5>
<div class="paragraph">
<p>QEMU replays support checkpointing, and this allows for a simplistic "reverse debugging" implementation since v5.2.0:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './linux/rand_check.out;./linux/poweroff.out;' --record
./run --eval-after './linux/rand_check.out;./linux/poweroff.out;' --replay --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>On another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>In GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>n
n
n
n
reverse-continue</pre>
</div>
</div>
<div class="paragraph">
<p>and we are back at <code>start_kernel</code></p>
</div>
<div class="paragraph">
<p><code>reverse-continue</code> proceeds to the latest of the earlier breakpoints or to the very beginning if there were no breakpoints before.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="qemu-trace-multicore"><a class="anchor" href="#qemu-trace-multicore"></a><a class="link" href="#qemu-trace-multicore">23.9.6. QEMU trace multicore</a></h4>
<div class="paragraph">
<p>TODO: is there any way to distinguish which instruction runs on each core? Doing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch x86_64 --cpus 2 --eval './linux/poweroff.out' --trace exec_tb
./qemu-trace2txt</pre>
</div>
</div>
<div class="paragraph">
<p>just appears to output both cores intertwined without any clear differentiation.</p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-get-guest-instruction-count"><a class="anchor" href="#qemu-get-guest-instruction-count"></a><a class="link" href="#qemu-get-guest-instruction-count">23.9.7. QEMU get guest instruction count</a></h4>
<div class="paragraph">
<p>TODO: <a href="https://stackoverflow.com/questions/58766571/how-to-count-the-number-of-guest-instructions-qemu-executed-from-the-beginning-t" class="bare">https://stackoverflow.com/questions/58766571/how-to-count-the-number-of-guest-instructions-qemu-executed-from-the-beginning-t</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-tracing"><a class="anchor" href="#gem5-tracing"></a><a class="link" href="#gem5-tracing">23.9.8. gem5 tracing</a></h4>
<div class="paragraph">
<p>gem5 provides also provides a tracing mechanism documented at: <a href="http://www.gem5.org/Trace_Based_Debugging" class="bare">http://www.gem5.org/Trace_Based_Debugging</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --eval 'm5 exit' --emulator gem5 --trace ExecAll
less "$(./getvar --arch aarch64 run_dir)/trace.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>Our wrapper just forwards the options to the <code>--debug-flags</code> gem5 option.</p>
</div>
<div class="paragraph">
<p>Keep in mind however that the disassembly is very broken in several places as of 2019q2, so you can&#8217;t always trust it.</p>
</div>
<div class="paragraph">
<p>Output the trace to stdout instead of a file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --eval 'm5 exit' \
  --trace ExecAll \
  --trace-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>We also have a shortcut for <code>--trace ExecAll -trace-stdout</code> with <code>--trace-insts-stdout</code></p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --eval 'm5 exit' \
  --trace-insts-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Be warned, the trace is humongous, at 16Gb.</p>
</div>
<div class="paragraph">
<p>This would produce a lot of output however, so you will likely not want that when tracing a Linux kernel boot instructions. But it can be very convenient for smaller traces such as <a href="#baremetal">Baremetal</a>.</p>
</div>
<div class="paragraph">
<p>List all available debug flags:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --gem5-exe-args='--debug-help' --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>but to understand most of them you have to look at the source code:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar gem5_source_dir)/src/cpu/SConscript"
less "$(./getvar gem5_source_dir)/src/cpu/exetrace.cc"</pre>
</div>
</div>
<div class="paragraph">
<p>The most important trace flags to know about are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-execall-trace-format"><code>ExecAll</code></a></p>
</li>
<li>
<p><code>Faults</code>: CPU exceptions / interrupts, see an example at: <a href="#arm-svc-instruction">ARM SVC instruction</a></p>
</li>
<li>
<p><a href="#gem5-registers-trace-format"><code>Registers</code></a></p>
</li>
<li>
<p><a href="#gem5-syscall-emulation-mode-syscall-tracing"><code>SyscallBase</code>, <code>SyscallVerbose</code></a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Trace internals are discussed at: <a href="#gem5-trace-internals">gem5 trace internals</a>.</p>
</div>
<div class="paragraph">
<p>As can be seen on the <code>Sconstruct</code>, <code>Exec</code> is just an alias that enables a set of flags.</p>
</div>
<div class="paragraph">
<p>We can make the trace smaller by naming the trace file as <code>trace.txt.gz</code>, which enables GZIP compression, but that is not currently exposed on our scripts, since you usually just need something human readable to work on.</p>
</div>
<div class="paragraph">
<p>Enabling tracing made the runtime about 4x slower on the <a href="#p51">2017 Lenovo ThinkPad P51</a>, with or without <code>.gz</code> compression.</p>
</div>
<div class="paragraph">
<p>Trace the source lines just like <a href="#trace-source-lines">for QEMU</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./trace-boot --arch aarch64 --emulator gem5
./trace2line --arch aarch64 --emulator gem5
less "$(./getvar --arch aarch64 run_dir)/trace-lines.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: 7452d399290c9c1fc6366cdad129ef442f323564 <code>./trace2line</code> this is too slow and takes hours. QEMU&#8217;s processing of 170k events takes 7 seconds. gem5&#8217;s processing is analogous, but there are 140M events, so it should take 7000 seconds ~ 2 hours which seems consistent with what I observe, so maybe there is no way to speed this up&#8230;&#8203; The workaround is to just use gem5&#8217;s <code>ExecSymbol</code> to get function granularity, and then GDB individually if line detail is needed?</p>
</div>
<div class="sect4">
<h5 id="gem5-trace-internals"><a class="anchor" href="#gem5-trace-internals"></a><a class="link" href="#gem5-trace-internals">23.9.8.1. gem5 trace internals</a></h5>
<div class="paragraph">
<p>gem5 traces are generated from <code>DPRINTF(&lt;trace-id&gt;</code> calls scattered throughout the code, except for <code>ExecAll</code> instruction traces, which uses <code>Debug::ExecEnable</code> directly..</p>
</div>
<div class="paragraph">
<p>The trace IDs are themselves encoded in <code>SConscript</code> files, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>DebugFlag('Event'</pre>
</div>
</div>
<div class="paragraph">
<p>in <code>src/cpu/SConscript</code>.</p>
</div>
<div class="paragraph">
<p>The build system then automatically adds the options to the <code>--debug-flags</code>.</p>
</div>
<div class="paragraph">
<p>For this entry, the build system then generates a file <code>build/ARM/debug/ExecEnable.hh</code>, which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>namespace Debug {
class SimpleFlag;
extern SimpleFlag ExecEnable;
}</pre>
</div>
</div>
<div class="paragraph">
<p>and must be included in from callers of <code>DPRINTF(</code> as <code>&lt;debug/ExecEnable.hh&gt;</code>.</p>
</div>
<div class="paragraph">
<p>Tested in b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-execall-trace-format"><a class="anchor" href="#gem5-execall-trace-format"></a><a class="link" href="#gem5-execall-trace-format">23.9.8.2. gem5 ExecAll trace format</a></h5>
<div class="paragraph">
<p>This debug flag traces all instructions.</p>
</div>
<div class="paragraph">
<p>The output format is of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>25007000: system.cpu T0 : @start_kernel    : stp
25007000: system.cpu T0 : @start_kernel.0  :   addxi_uop   ureg0, sp, #-112 : IntAlu :  D=0xffffff8008913f90
25007500: system.cpu T0 : @start_kernel.1  :   strxi_uop   x29, [ureg0] : MemWrite :  D=0x0000000000000000 A=0xffffff8008913f90
25008000: system.cpu T0 : @start_kernel.2  :   strxi_uop   x30, [ureg0, #8] : MemWrite :  D=0x0000000000000000 A=0xffffff8008913f98
25008500: system.cpu T0 : @start_kernel.3  :   addxi_uop   sp, ureg0, #0 : IntAlu :  D=0xffffff8008913f90</pre>
</div>
</div>
<div class="paragraph">
<p>There are two types of lines:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>full instructions, as the first line. Only shown if the <code>ExecMacro</code> flag is given.</p>
</li>
<li>
<p>micro ops that constitute the instruction, the lines that follow. Yes, <code>aarch64</code> also has microops: <a href="https://superuser.com/questions/934752/do-arm-processors-like-cortex-a9-use-microcode/934755#934755" class="bare">https://superuser.com/questions/934752/do-arm-processors-like-cortex-a9-use-microcode/934755#934755</a>. Only shown if the <code>ExecMicro</code> flag is given.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Breakdown:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>25007500</code>: time count in some unit. Note how the microops execute at further timestamps.</p>
</li>
<li>
<p><code>system.cpu</code>: distinguishes between CPUs when there are more than one. For example, running <a href="#arm-baremetal-multicore">Section 33.10.3, &#8220;ARM baremetal multicore&#8221;</a> with two cores produces <code>system.cpu0</code> and <code>system.cpu1</code></p>
</li>
<li>
<p><code>T0</code>: thread number. TODO: <a href="https://superuser.com/questions/133082/hyper-threading-and-dual-core-whats-the-difference/995858#995858">hyperthread</a>? How to play with it?</p>
<div class="paragraph">
<p><code>config</code>.ini has <code>--param 'system.multi_thread = True' --param 'system.cpu[0].numThreads = 2'</code>, but in <a href="#arm-baremetal-multicore">ARM baremetal multicore</a> the first one alone does not produce <code>T1</code>, and with the second one simulation blows up with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fatal: fatal condition interrupts.size() != numThreads occurred: CPU system.cpu has 1 interrupt controllers, but is expecting one per thread (2)</pre>
</div>
</div>
</li>
<li>
<p><code>@start_kernel</code>: we are in the <code>start_kernel</code> function. Awesome feature! Implemented with libelf <a href="https://sourceforge.net/projects/elftoolchain/" class="bare">https://sourceforge.net/projects/elftoolchain/</a> copy pasted in-tree <code>ext/libelf</code>. To get raw addresses, remove the <code>ExecSymbol</code>, which is enabled by <code>Exec</code>. This can be done with <code>Exec,-ExecSymbol</code>.</p>
</li>
<li>
<p><code>.1</code> as in <code>@start_kernel.1</code>: index of the <a href="#gem5-microops">gem5 microops</a></p>
</li>
<li>
<p><code>stp</code>: instruction disassembly. Note however that the disassembly of many instructions are very broken as of 2019q2, and you can&#8217;t just trust them blindly.</p>
</li>
<li>
<p><code>strxi_uop   x29, [ureg0]</code>: microop disassembly.</p>
</li>
<li>
<p><code>MemWrite :  D=0x0000000000000000 A=0xffffff8008913f90</code>: a memory write microop:</p>
<div class="ulist">
<ul>
<li>
<p><code>D</code> stands for data, and represents the value that was written to memory or to a register</p>
</li>
<li>
<p><code>A</code> stands for address, and represents the address to which the value was written. It only shows when data is being written to memory, but not to registers.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The best way to verify all of this is to write some <a href="#baremetal">baremetal code</a></p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-registers-trace-format"><a class="anchor" href="#gem5-registers-trace-format"></a><a class="link" href="#gem5-registers-trace-format">23.9.8.3. gem5 Registers trace format</a></h5>
<div class="paragraph">
<p>This flag shows a more detailed register usage than <a href="#gem5-execall-trace-format">gem5 ExecAll trace format</a>.</p>
</div>
<div class="paragraph">
<p>For example, if we run in LKMC 0323e81bff1d55b978a4b36b9701570b59b981eb:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/arch/aarch64/add.S --emulator gem5 --trace ExecAll,Registers --trace-stdout</pre>
</div>
</div>
<div class="paragraph">
<p>then the stdout contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  31000: system.cpu A0 T0 : @main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
  31500: system.cpu.[tid:0]: Setting int reg 34 (34) to 0.
  31500: system.cpu.[tid:0]: Reading int reg 0 (0) as 0x1.
  31500: system.cpu.[tid:0]: Setting int reg 1 (1) to 0x3.
  31500: system.cpu A0 T0 : @main_after_prologue+4    :   add   x1, x0, #2         : IntAlu :  D=0x0000000000000003  flags=(IsInteger)
  32000: system.cpu.[tid:0]: Setting int reg 34 (34) to 0.
  32000: system.cpu.[tid:0]: Reading int reg 1 (1) as 0x3.
  32000: system.cpu.[tid:0]: Reading int reg 31 (34) as 0.
  32000: system.cpu.[tid:0]: Setting int reg 0 (0) to 0x3.</pre>
</div>
</div>
<div class="paragraph">
<p>which corresponds to the two following instructions:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov x0, 1
add x1, x0, 2</pre>
</div>
</div>
<div class="paragraph">
<p>TODO that format is either buggy or is very difficult to understand:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>what is <code>34</code>? Presumably some flags register?</p>
</li>
<li>
<p>what do the numbers in parenthesis mean at <code>31 (34)</code>? Presumably some flags register?</p>
</li>
<li>
<p>why is the first instruction setting <code>reg 1</code> and the second one <code>reg 0</code>, given that the first sets <code>x0</code> and the second <code>x1</code>?</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="gem5-tarmac-traces"><a class="anchor" href="#gem5-tarmac-traces"></a><a class="link" href="#gem5-tarmac-traces">23.9.8.4. gem5 TARMAC traces</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/54882466/how-to-use-the-tarmac-tracer-with-gem5" class="bare">https://stackoverflow.com/questions/54882466/how-to-use-the-tarmac-tracer-with-gem5</a></p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-tracing-internals"><a class="anchor" href="#gem5-tracing-internals"></a><a class="link" href="#gem5-tracing-internals">23.9.8.5. gem5 tracing internals</a></h5>
<div class="paragraph">
<p>As of gem5 16eeee5356585441a49d05c78abc328ef09f7ace the default tracer is <code>ExeTracer</code>. It is set at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/cpu/BaseCPU.py:63:default_tracer = ExeTracer()</pre>
</div>
</div>
<div class="paragraph">
<p>which then gets used at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class BaseCPU(ClockedObject):
    [...]
    tracer = Param.InstTracer(default_tracer, "Instruction tracer")</pre>
</div>
</div>
<div class="paragraph">
<p>All tracers derive from the common <code>InstTracer</code> base class:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git grep ': InstTracer'</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/arch/arm/tracers/tarmac_parser.hh:218:    TarmacParser(const Params *p) : InstTracer(p), startPc(p-&gt;start_pc),
src/arch/arm/tracers/tarmac_tracer.cc:57:  : InstTracer(p),
src/cpu/exetrace.hh:67:    ExeTracer(const Params *params) : InstTracer(params)
src/cpu/inst_pb_trace.cc:72:    : InstTracer(p), buf(nullptr), bufSize(0), curMsg(nullptr)
src/cpu/inteltrace.hh:63:    IntelTrace(const IntelTraceParams *p) : InstTracer(p)</pre>
</div>
</div>
<div class="paragraph">
<p>As mentioned at <a href="#gem5-tarmac-traces">gem5 TARMAC traces</a>, there appears to be no way to select those currently without hacking the config scripts.</p>
</div>
<div class="paragraph">
<p>TARMAC is described at: <a href="#gem5-tarmac-traces">gem5 TARMAC traces</a>.</p>
</div>
<div class="paragraph">
<p>TODO: are <code>IntelTrace</code> and <code>TarmacParser</code> useful for anything or just relics?</p>
</div>
<div class="paragraph">
<p>Then there is also the <code>NativeTrace</code> class:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/cpu/nativetrace.hh:68:class NativeTrace : public ExeTracer</pre>
</div>
</div>
<div class="paragraph">
<p>which gets implemented in a few different ISAs, but not all:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/arch/arm/nativetrace.hh:40:class ArmNativeTrace : public NativeTrace
src/arch/sparc/nativetrace.hh:41:class SparcNativeTrace : public NativeTrace
src/arch/x86/nativetrace.hh:41:class X86NativeTrace : public NativeTrace</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: I can&#8217;t find any usages of those classes from in-tree configs.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="qemu-gui-is-unresponsive"><a class="anchor" href="#qemu-gui-is-unresponsive"></a><a class="link" href="#qemu-gui-is-unresponsive">23.10. QEMU GUI is unresponsive</a></h3>
<div class="paragraph">
<p>Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.</p>
</div>
<div class="paragraph">
<p>We have not managed to track this problem down yet, but the following workaround always works:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-Shift-U
Ctrl-C
root</pre>
</div>
</div>
<div class="paragraph">
<p>This started happening when we switched to building QEMU through Buildroot, and has not been observed on later Ubuntu.</p>
</div>
<div class="paragraph">
<p>Using text mode is another workaround if you don&#8217;t need GUI features.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="gem5"><a class="anchor" href="#gem5"></a><a class="link" href="#gem5">24. gem5</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Getting started at: <a href="#gem5-buildroot-setup">Section 2.4, &#8220;gem5 Buildroot setup&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>gem5 has a bunch of crappiness, mostly described at: <a href="#gem5-vs-qemu">gem5 vs QEMU</a>, but it does deserve some credit on the following points:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>insanely configurable system topology from Python without recompiling, made possible in part due to a well defined memory packet structure that allows adding caches and buses transparently</p>
</li>
<li>
<p>each micro architectural model (<a href="#gem5-cpu-types">gem5 CPU types</a>) works with all ISAs</p>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="gem5-vs-qemu"><a class="anchor" href="#gem5-vs-qemu"></a><a class="link" href="#gem5-vs-qemu">24.1. gem5 vs QEMU</a></h3>
<div class="ulist">
<ul>
<li>
<p>advantages of gem5:</p>
<div class="ulist">
<ul>
<li>
<p>simulates a generic more realistic <a href="#gem5-cpu-types">optionally pipelined and out-of-order</a> CPU cycle by cycle, including a realistic DRAM memory access model with latencies, caches and page table manipulations. This allows us to:</p>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>do much more realistic performance benchmarking with it, which makes absolutely no sense in QEMU, which is purely functional</p>
</li>
<li>
<p>make certain functional observations that are not possible in QEMU, e.g.:</p>
<div class="ulist">
<ul>
<li>
<p>use Linux kernel APIs that flush cache memory like DMA, which are crucial for driver development. In QEMU, the driver would still work even if we forget to flush caches.</p>
</li>
<li>
<p>spectre / meltdown:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15319.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15319.html</a></p>
</li>
<li>
<p><a href="https://github.com/jlpresearch/gem5/tree/spectre-test" class="bare">https://github.com/jlpresearch/gem5/tree/spectre-test</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>It is not of course truly cycle accurate, as that:</p>
</div>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>would require exposing proprietary information of the CPU designs: <a href="https://stackoverflow.com/questions/17454955/can-you-check-performance-of-a-program-running-with-qemu-simulator/33580850#33580850" class="bare">https://stackoverflow.com/questions/17454955/can-you-check-performance-of-a-program-running-with-qemu-simulator/33580850#33580850</a></p>
</li>
<li>
<p>would make the simulation even slower TODO confirm, by how much</p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>but the approximation is reasonable.</p>
</div>
<div class="paragraph">
<p>It is used mostly for microarchitecture research purposes: when you are making a new chip technology, you don&#8217;t really need to specialize enormously to an existing microarchitecture, but rather develop something that will work with a wide range of future architectures.</p>
</div>
</li>
<li>
<p>runs are deterministic by default, unlike QEMU which has a special <a href="#qemu-record-and-replay">QEMU record and replay</a> mode, that requires first playing the content once and then replaying</p>
</li>
<li>
<p>gem5 ARM at least appears to implement more low level CPU functionality than QEMU, e.g. QEMU only added EL2 in 2018: <a href="https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up" class="bare">https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up</a> See also: <a href="#arm-exception-levels">Section 33.10.1, &#8220;ARM exception levels&#8221;</a></p>
</li>
<li>
<p>gem5 offers more advanced logging, even for non micro architectural things which QEMU models in some way, e.g. <a href="#qemu-trace-memory-accesses">QEMU trace memory accesses</a>, because QEMU&#8217;s binary translation optimizations reduce visibility</p>
</li>
</ul>
</div>
</li>
<li>
<p>disadvantages of gem5:</p>
<div class="ulist">
<ul>
<li>
<p>slower than QEMU, see: <a href="#benchmark-linux-kernel-boot">Section 35.2.1, &#8220;Benchmark Linux kernel boot&#8221;</a></p>
<div class="paragraph">
<p>This implies that the user base is much smaller, since no Android devs.</p>
</div>
<div class="paragraph">
<p>Instead, we have only chip makers, who keep everything that really works closed, and researchers, who can&#8217;t version track or document code properly &gt;:-) And this implies that:</p>
</div>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>the documentation is more scarce</p>
</li>
<li>
<p>it takes longer to support new hardware features</p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>Well, not that AOSP is that much better anyway.</p>
</div>
</li>
<li>
<p>not sure: gem5 has BSD license while QEMU has GPL</p>
<div class="paragraph">
<p>This suits chip makers that want to distribute forks with secret IP to their customers.</p>
</div>
<div class="paragraph">
<p>On the other hand, the chip makers tend to upstream less, and the project becomes more crappy in average :-)</p>
</div>
</li>
<li>
<p>gem5 is way more complex and harder to modify and maintain</p>
<div class="paragraph">
<p>The only hairy thing in QEMU is the binary code generation.</p>
</div>
<div class="paragraph">
<p>gem5 however has tended towards horrendous intensive <a href="#gem5-code-generation">code generation</a> in order to support all its different hardware types</p>
</div>
<div class="paragraph">
<p>gem5 also has a complex Python interface which is also largely auto-generated, which greatly increases the maintenance complexity of the project: <a href="#embedding-python-in-another-application">Embedding Python in another application</a>.</p>
</div>
<div class="paragraph">
<p>This is done so that reconfiguring platforms can be done quickly without recompiling, and it is amazing when it works, but the maintenance costs are also very high. For example, <a href="#pybind11">pybind11</a> of several trivial <code>param_</code> files accounted for 50% of the build time at one point: <a href="#pybind11-accounts-for-50-of-gem5-build-time">pybind11 accounts for 50% of gem5 build time</a>.</p>
</div>
<div class="paragraph">
<p>All of this also makes it hard to setup an IDE for developing gem5: <a href="#gem5-eclipse-configuration">gem5 Eclipse configuration</a></p>
</div>
<div class="paragraph">
<p>The feelings of helplessness this brings are well summarized by the following CSDN article <a href="https://blog.csdn.net/maokelong95/article/details/85333905" class="bare">https://blog.csdn.net/maokelong95/article/details/85333905</a>:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Found DPRINTF based debugging unable to meet your needs?</p>
</div>
<div class="paragraph">
<p>Found GDB based debugging unfriendly to human beings?</p>
</div>
<div class="paragraph">
<p>Want to debug gem5 source with the help of modern IDEs like Eclipse?</p>
</div>
<div class="paragraph">
<p>Failed in getting help from GEM5 community?</p>
</div>
<div class="paragraph">
<p>Come on, dude! Here is the up-to-date tutorial for you!</p>
</div>
<div class="paragraph">
<p>Just be ready for THE ENDLESS NIGHTMARE gem5 will bring!</p>
</div>
</blockquote>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="gem5-run-benchmark"><a class="anchor" href="#gem5-run-benchmark"></a><a class="link" href="#gem5-run-benchmark">24.2. gem5 run benchmark</a></h3>
<div class="paragraph">
<p>OK, this is why we used gem5 in the first place, performance measurements!</p>
</div>
<div class="paragraph">
<p>Let&#8217;s see how many cycles dhrystone, which Buildroot provides, takes for a few different input parameters.</p>
</div>
<div class="paragraph">
<p>We will do that for various input parameters on full system by taking a checkpoint after the boot finishes a fast atomic CPU boot, and then we will restore in a more detailed mode and run the benchmark:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_DHRYSTONE=y'
# Boot fast, take checkpoint, and exit.
./run --arch aarch64 --emulator gem5 --eval-after './gem5.sh'

# Restore the checkpoint after boot, and benchmark with input 1000.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --eval-after './gem5.sh' \
  --gem5-readfile 'm5 resetstats;dhrystone 1000;m5 dumpstats' \
  --gem5-restore 1 \
  -- \
  --cpu-type=HPI \
  --restore-with-cpu=HPI \
  --caches \
  --l2cache \
  --l1d_size=64kB \
  --l1i_size=64kB \
  --l2_size=256kB \
;
# Get the value for number of cycles.
# head because there are two lines: our dumpstats and the
# automatic dumpstats at the end which we don't care about.
./gem5-stat --arch aarch64 | head -n 1

# Now for input 10000.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --eval-after './gem5.sh' \
  --gem5-readfile 'm5 resetstats;dhrystone 10000;m5 dumpstats' \
  --gem5-restore 1 \
  -- \
  --cpu-type=HPI \
  --restore-with-cpu=HPI \
  --caches \
  --l2cache \
  --l1d_size=64kB \
  --l1i_size=64kB \
  --l2_size=256kB \
;
./gem5-stat --arch aarch64 | head -n 1</pre>
</div>
</div>
<div class="paragraph">
<p>If you ever need a shell to quickly inspect the system state after boot, you can just use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --eval-after './gem5.sh' \
  --gem5-readfile 'sh' \
  --gem5-restore 1 \</pre>
</div>
</div>
<div class="paragraph">
<p>This procedure is further automated and DRYed up at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-bench-dhrystone
cat out/gem5-bench-dhrystone.txt</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/gem5-bench-dhrystone">gem5-bench-dhrystone</a></p>
</div>
<div class="paragraph">
<p>Output at 2438410c25e200d9766c8c65773ee7469b599e4a + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>n cycles
1000 13665219
10000 20559002
100000 85977065</pre>
</div>
</div>
<div class="paragraph">
<p>so as expected, the Dhrystone run with a larger input parameter <code>100000</code> took more cycles than the ones with smaller input parameters.</p>
</div>
<div class="paragraph">
<p>The <code>gem5-stats</code> commands output the approximate number of CPU cycles it took Dhrystone to run.</p>
</div>
<div class="paragraph">
<p>A more naive and simpler to understand approach would be a direct:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --eval 'm5 checkpoint;m5 resetstats;dhrystone 10000;m5 exit'</pre>
</div>
</div>
<div class="paragraph">
<p>but the problem is that this method does not allow to easily run a different script without running the boot again. The <code>./gem5.sh</code> script works around that by using <a href="#m5-readfile">m5 readfile</a> as explained further at: <a href="#gem5-restore-new-script">Section 24.6.3, &#8220;gem5 checkpoint restore and run a different script&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Now you can play a fun little game with your friends:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>pick a computational problem</p>
</li>
<li>
<p>make a program that solves the computation problem, and outputs output to stdout</p>
</li>
<li>
<p>write the code that runs the correct computation in the smallest number of cycles possible</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Interesting algorithms and benchmarks for this game are being collected at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#algorithms">Algorithms</a></p>
</li>
<li>
<p><a href="#benchmarks">Benchmarks</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To find out why your program is slow, a good first step is to have a look at the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
</div>
<div class="sect3">
<h4 id="skip-extra-benchmark-instructions"><a class="anchor" href="#skip-extra-benchmark-instructions"></a><a class="link" href="#skip-extra-benchmark-instructions">24.2.1. Skip extra benchmark instructions</a></h4>
<div class="paragraph">
<p>A few imperfections of our <a href="#gem5-run-benchmark">benchmarking method</a> are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>when we do <code>m5 resetstats</code> and <code>m5 exit</code>, there is some time passed before the <code>exec</code> system call returns and the actual benchmark starts and ends</p>
</li>
<li>
<p>the benchmark outputs to stdout, which means so extra cycles in addition to the actual computation. But TODO: how to get the output to check that it is correct without such IO cycles?</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Solutions to these problems include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>modify benchmark code with instrumentation directly, see <a href="#m5ops-instructions">m5ops instructions</a> for an example.</p>
</li>
<li>
<p>monitor known addresses TODO possible? Create an example.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Discussion at: <a href="https://stackoverflow.com/questions/48944587/how-to-count-the-number-of-cpu-clock-cycles-between-the-start-and-end-of-a-bench/48944588#48944588" class="bare">https://stackoverflow.com/questions/48944587/how-to-count-the-number-of-cpu-clock-cycles-between-the-start-and-end-of-a-bench/48944588#48944588</a></p>
</div>
<div class="paragraph">
<p>Those problems should be insignificant if the benchmark runs for long enough however.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-system-parameters"><a class="anchor" href="#gem5-system-parameters"></a><a class="link" href="#gem5-system-parameters">24.3. gem5 system parameters</a></h3>
<div class="paragraph">
<p>Besides optimizing a program for a given CPU setup, chip developers can also do the inverse, and optimize the chip for a given benchmark!</p>
</div>
<div class="paragraph">
<p>The rabbit hole is likely deep, but let&#8217;s scratch a bit of the surface.</p>
</div>
<div class="sect3">
<h4 id="number-of-cores"><a class="anchor" href="#number-of-cores"></a><a class="link" href="#number-of-cores">24.3.1. Number of cores</a></h4>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --cpus 2 --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Can be checked with <code>/proc/cpuinfo</code> or <a href="#sysconf">getconf</a> in Ubuntu 18.04:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/cpuinfo
getconf _NPROCESSORS_CONF</pre>
</div>
</div>
<div class="paragraph">
<p>Or from <a href="#user-mode-simulation">User mode simulation</a>, we can use either of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#sysconf">sysconf</a> with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sysconf.c">userland/linux/sysconf.c</a></p>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --emulator gem5 --userland userland/linux/sysconf.c | grep _SC_NPROCESSORS_ONLN</pre>
</div>
</div>
</li>
<li>
<p><a href="#cpp-multithreading">C++ multithreading</a>'s <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/thread_hardware_concurrency.cpp">userland/cpp/thread_hardware_concurrency.cpp</a>:</p>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --emulator gem5 --userland userland/cpp/thread_hardware_concurrency.cpp</pre>
</div>
</div>
</li>
<li>
<p>direct access to several special filesystem files that contain this information e.g. via <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/cat.c">userland/c/cat.c</a>:</p>
<div class="literalblock">
<div class="content">
<pre>./run --cpus 2 --emulator gem5 --userland userland/c/cat.c --cli-args /proc/cpuinfo</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="qemu-user-mode-multithreading"><a class="anchor" href="#qemu-user-mode-multithreading"></a><a class="link" href="#qemu-user-mode-multithreading">24.3.1.1. QEMU user mode multithreading</a></h5>
<div class="paragraph">
<p><a href="#user-mode-simulation">User mode simulation</a> QEMU v4.0.0 always shows the number of cores of the host, presumably because the thread switching uses host threads directly which would make that harder to implement.</p>
</div>
<div class="paragraph">
<p>It does not seem possible to make the guest see a different number of cores than what the host has. Full system does have the <code>-smp</code> options, which controls this.</p>
</div>
<div class="paragraph">
<p>E.g., all of of the following output the same as <code>nproc</code> on the host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>nproc
./run --cpus 1 --userland userland/cpp/thread_hardware_concurrency.cpp
./run --cpus 2 --userland userland/cpp/thread_hardware_concurrency.cpp</pre>
</div>
</div>
<div class="paragraph">
<p>This random page suggests that QEMU splits one host thread thread per guest thread, and thus presumably delegates context switching to the host kernel: <a href="https://qemu.weilnetz.de/w64/2012/2012-12-04/qemu-tech.html#User-emulation-specific-details" class="bare">https://qemu.weilnetz.de/w64/2012/2012-12-04/qemu-tech.html#User-emulation-specific-details</a></p>
</div>
<div class="paragraph">
<p>We can confirm that with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland userland/posix/pthread_count.c --cli-args 4
ps Haux | grep qemu | wc</pre>
</div>
</div>
<div class="paragraph">
<p>Remember <a href="#qemu-user-mode-does-not-show-stdout-immediately">QEMU user mode does not show stdout immediately</a> though.</p>
</div>
<div class="paragraph">
<p>At 369a47fc6e5c2f4a7f911c1c058b6088f8824463 + 1 QEMU appears to spawn 3 host threads plus one for every new guest thread created. Remember that <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_count.c">userland/posix/pthread_count.c</a> spawns N + 1 total threads if you count the <code>main</code> thread.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-arm-full-system-with-more-than-8-cores"><a class="anchor" href="#gem5-arm-full-system-with-more-than-8-cores"></a><a class="link" href="#gem5-arm-full-system-with-more-than-8-cores">24.3.1.2. gem5 ARM full system with more than 8 cores</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/50248067/how-to-run-a-gem5-arm-aarch64-full-system-simulation-with-fs-py-with-more-than-8" class="bare">https://stackoverflow.com/questions/50248067/how-to-run-a-gem5-arm-aarch64-full-system-simulation-with-fs-py-with-more-than-8</a></p>
</div>
<div class="paragraph">
<p>With <a href="#arm-gic">GICv3</a>, tested at LKMC 224fae82e1a79d9551b941b19196c7e337663f22 gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 on vanilla kernel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --cpus 16 \
  -- \
  --machine-type VExpress_GEM5_V2 \
;</pre>
</div>
</div>
<div class="paragraph">
<p>boots to a shell and <code>nproc</code> shows <code>16</code>.</p>
</div>
<div class="paragraph">
<p>For the GICv2 extension method, build the kernel with the <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a>, and then run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --linux-build-id gem5-v4.15 \
  --emulator gem5 \
  --cpus 16 \
  -- \
  --param 'system.realview.gic.gem5_extensions = True' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Tested in LKMC 788087c6f409b84adf3cff7ac050fa37df6d4c46. It fails after boot with <code>FATAL: kernel too old</code> as mentioned at: <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a> but everything seems to work on the gem5 side of things.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-cache-size"><a class="anchor" href="#gem5-cache-size"></a><a class="link" href="#gem5-cache-size">24.3.2. gem5 cache size</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/49624061/how-to-run-gem5-simulator-in-fs-mode-without-cache/49634544#49634544" class="bare">https://stackoverflow.com/questions/49624061/how-to-run-gem5-simulator-in-fs-mode-without-cache/49634544#49634544</a></p>
</div>
<div class="paragraph">
<p>A quick <code>./run --emulator gem5 -- -h</code> leads us to the options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--caches
--l1d_size=1024
--l1i_size=1024
--l2cache
--l2_size=1024
--l3_size=1024</pre>
</div>
</div>
<div class="paragraph">
<p>But keep in mind that it only affects benchmark performance of the most detailed CPU types as shown at: <a href="#table-gem5-cache-cpu-type">Table 2, &#8220;gem5 cache support in function of CPU type&#8221;</a>.</p>
</div>
<table id="table-gem5-cache-cpu-type" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 2. gem5 cache support in function of CPU type</caption>
<colgroup>
<col style="width: 33.3333%;">
<col style="width: 33.3333%;">
<col style="width: 33.3334%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">arch</th>
<th class="tableblock halign-left valign-top">CPU type</th>
<th class="tableblock halign-left valign-top">caches used</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">X86</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>AtomicSimpleCPU</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">X86</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>DerivO3CPU</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">?*</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ARM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>AtomicSimpleCPU</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">no</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">ARM</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>HPI</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">yes</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>*: couldn&#8217;t test because of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/49011096/how-to-switch-cpu-models-in-gem5-after-restoring-a-checkpoint-and-then-observe-t" class="bare">https://stackoverflow.com/questions/49011096/how-to-switch-cpu-models-in-gem5-after-restoring-a-checkpoint-and-then-observe-t</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Cache sizes can in theory be checked with the methods described at: <a href="https://superuser.com/questions/55776/finding-l2-cache-size-in-linux" class="bare">https://superuser.com/questions/55776/finding-l2-cache-size-in-linux</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lscpu
cat /sys/devices/system/cpu/cpu0/cache/index2/size</pre>
</div>
</div>
<div class="paragraph">
<p>and on Ubuntu 20.04 host <a href="#sysconf">but not Buildroot 1.31.1</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>getconf -a | grep CACHE</pre>
</div>
</div>
<div class="paragraph">
<p>and we also have an easy to use userland executable using <a href="#sysconf">sysconf</a> at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sysconf.c">userland/linux/sysconf.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --userland userland/linux/sysconf.c</pre>
</div>
</div>
<div class="paragraph">
<p>but for some reason the Linux kernel is not seeing the cache sizes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/49008792/why-doesnt-the-linux-kernel-see-the-cache-sizes-in-the-gem5-emulator-in-full-sy" class="bare">https://stackoverflow.com/questions/49008792/why-doesnt-the-linux-kernel-see-the-cache-sizes-in-the-gem5-emulator-in-full-sy</a></p>
</li>
<li>
<p><a href="http://gem5-users.gem5.narkive.com/4xVBlf3c/verify-cache-configuration" class="bare">http://gem5-users.gem5.narkive.com/4xVBlf3c/verify-cache-configuration</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Behaviour breakdown:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>arm QEMU and gem5 (both <code>AtomicSimpleCPU</code> or <code>HPI</code>), x86 gem5: <code>/sys</code> files don&#8217;t exist, and <code>getconf</code> and <code>lscpu</code> value empty</p>
</li>
<li>
<p>x86 QEMU: <code>/sys</code> files exist, but <code>getconf</code> and <code>lscpu</code> values still empty</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The only precise option is therefore to look at <a href="#gem5-config-ini">gem5 config.ini</a> as done at: <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a>.</p>
</div>
<div class="paragraph">
<p>Or for a quick and dirty performance measurement approach instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-bench-cache -- --arch aarch64
cat "$(./getvar --arch aarch64 run_dir)/bench-cache.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 1000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024   --l1i_size=1024   --l2_size=1024   --l3_size=1024   --cpu-type=HPI --restore-with-cpu=HPI
time 23.82
exit_status 0
cycles 93284622
instructions 4393457

cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 1000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB --cpu-type=HPI --restore-with-cpu=HPI
time 14.91
exit_status 0
cycles 10128985
instructions 4211458

cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 10000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024   --l1i_size=1024   --l2_size=1024   --l3_size=1024   --cpu-type=HPI --restore-with-cpu=HPI
time 51.87
exit_status 0
cycles 188803630
instructions 12401336

cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 10000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB --cpu-type=HPI --restore-with-cpu=HPI
time 35.35
exit_status 0
cycles 20715757
instructions 12192527

cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 100000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024   --l1i_size=1024   --l2_size=1024   --l3_size=1024   --cpu-type=HPI --restore-with-cpu=HPI
time 339.07
exit_status 0
cycles 1176559936
instructions 94222791

cmd ./run --emulator gem5 --arch aarch64 --gem5-readfile "dhrystone 100000" --gem5-restore 1 -- --caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB --cpu-type=HPI --restore-with-cpu=HPI
time 240.37
exit_status 0
cycles 125666679
instructions 91738770</pre>
</div>
</div>
<div class="paragraph">
<p>We make the following conclusions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the number of instructions almost does not change: the CPU is waiting for memory all the extra time. TODO: why does it change at all?</p>
</li>
<li>
<p>the wall clock execution time is not directionally proportional to the number of cycles: here we had a 10x cycle increase, but only 2x time increase. This suggests that the simulation of cycles in which the CPU is waiting for memory to come back is faster.</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gem5-dram-model"><a class="anchor" href="#gem5-dram-model"></a><a class="link" href="#gem5-dram-model">24.3.3. gem5 DRAM model</a></h4>
<div class="paragraph">
<p>Some info at: <a href="#timingsimplecpu-analysis-1">TimingSimpleCPU analysis #1</a> but highly TODO :-)</p>
</div>
<div class="sect4">
<h5 id="gem5-memory-latency"><a class="anchor" href="#gem5-memory-latency"></a><a class="link" href="#gem5-memory-latency">24.3.3.1. gem5 memory latency</a></h5>
<div class="paragraph">
<p>TODO These look promising:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--list-mem-types
--mem-type=MEM_TYPE
--mem-channels=MEM_CHANNELS
--mem-ranks=MEM_RANKS
--mem-size=MEM_SIZE</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: now to verify this with the Linux kernel? Besides raw performance benchmarks.</p>
</div>
<div class="paragraph">
<p>Now for a raw simplistic benchmark on <a href="#gem5-timingsimplecpu"><code>TimingSimpleCPU</code></a> without caches via <a href="#c-busy-loop">C busy loop</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --cli-args 1000000 --emulator gem5 --userland userland/gcc/busy_loop.c -- --cpu-type TimingSimpleCPU</pre>
</div>
</div>
<div class="paragraph">
<p>LKMC eb22fd3b6e7fff7e9ef946a88b208debf5b419d5 gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Exiting @ tick 897173931000 because exiting with last active thread context</pre>
</div>
</div>
<div class="paragraph">
<p>and now because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>we have no caches, each instruction is fetched from memory</p>
</li>
<li>
<p>each loop contains 11 instructions as shown at <a href="#c-busy-loop">Section 36.2, &#8220;C busy loop&#8221;</a></p>
</li>
<li>
<p>and supposing that the loop dominated executable pre/post <code>main</code>, which we know is true since as shown in <a href="#benchmark-emulators-on-userland-executables">Benchmark emulators on userland executables</a> an empty dynamically linked C program only as about 100k instructions, while our loop runs 1000000 * 11 = 12M.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>we should have about 1000000 * 11 / 897173931000 ps ~ 12260722 ~ 12MB/s of random accesses. The default memory type used is <code>DDR3_1600_8x8</code> as per:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>common/Options.py:101:    parser.add_option("--mem-type", type="choice", default="DDR3_1600_8x8</pre>
</div>
</div>
<div class="paragraph">
<p>and according to <a href="https://en.wikipedia.org/wiki/DDR3_SDRAM" class="bare">https://en.wikipedia.org/wiki/DDR3_SDRAM</a> that reaches 6400 MB/s so we are only off by a factor of 50x :-) TODO. Maybe if the minimum transaction if 64 bytes, we would be on point.</p>
</div>
<div class="paragraph">
<p>Another example we could use later on is <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a>, but then that mixes icache and dcache accesses, so the analysis is a bit more complex:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --cli-args 0x1000000 --emulator gem5 --userland userland/gcc/busy_loop.c -- --cpu-type TimingSimpleCPU</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="memory-size"><a class="anchor" href="#memory-size"></a><a class="link" href="#memory-size">24.3.3.2. Memory size</a></h5>
<div class="paragraph">
<p>Can be set across emulators with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --memory 512M</pre>
</div>
</div>
<div class="paragraph">
<p>We can verify this on the guest directly from the kernel with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/meminfo</pre>
</div>
</div>
<div class="paragraph">
<p>as of LKMC 1e969e832f66cb5a72d12d57c53fb09e9721d589 this output contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>MemTotal:         498472 kB</pre>
</div>
</div>
<div class="paragraph">
<p>which we expand with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '0x%X\n' $((498472 * 1024))</pre>
</div>
</div>
<div class="paragraph">
<p>to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x1E6CA000</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: why is this value a bit smaller than 512M?</p>
</div>
<div class="paragraph">
<p><code>free</code> also gives the same result:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>free -b</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>             total       used       free     shared    buffers     cached
Mem:     510435328   20385792  490049536          0     503808    2760704
-/+ buffers/cache:   17121280  493314048
Swap:            0          0          0</pre>
</div>
</div>
<div class="paragraph">
<p>which we expand with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '0x%X\n' 510435328$((498472 * 1024)</pre>
</div>
</div>
<div class="paragraph">
<p><code>man free</code> from Ubuntu&#8217;s procps 3.3.15 tells us that <code>free</code> obtains this information from <code>/proc/meminfo</code> as well.</p>
</div>
<div class="paragraph">
<p>From C, we can get this information with <code>sysconf(_SC_PHYS_PAGES)</code> or <code>get_phys_pages()</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./linux/total_memory.out</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/total_memory.c">userland/linux/total_memory.c</a></p>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE) = 0x1E6CA000
sysconf(_SC_AVPHYS_PAGES) * sysconf(_SC_PAGESIZE) = 0x1D178000
get_phys_pages() * sysconf(_SC_PAGESIZE) = 0x1E6CA000
get_avphys_pages() * sysconf(_SC_PAGESIZE) = 0x1D178000</pre>
</div>
</div>
<div class="paragraph">
<p>This is mentioned at: <a href="https://stackoverflow.com/questions/22670257/getting-ram-size-in-c-linux-non-precise-result/22670407#22670407" class="bare">https://stackoverflow.com/questions/22670257/getting-ram-size-in-c-linux-non-precise-result/22670407#22670407</a></p>
</div>
<div class="paragraph">
<p>AV means available and gives the free memory: <a href="https://stackoverflow.com/questions/14386856/c-check-available-ram/57659190#57659190" class="bare">https://stackoverflow.com/questions/14386856/c-check-available-ram/57659190#57659190</a></p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-dram-setup"><a class="anchor" href="#gem5-dram-setup"></a><a class="link" href="#gem5-dram-setup">24.3.3.3. gem5 DRAM setup</a></h5>
<div class="paragraph">
<p>This can be explored pretty well from <a href="#gem5-config-ini">gem5 config.ini</a>.</p>
</div>
<div class="paragraph">
<p>se.py just has a single <code>DDR3_1600_8x8</code> DRAM with size given as <a href="#memory-size">Memory size</a> and physical address starting at 0.</p>
</div>
<div class="paragraph">
<p>fs.py also has that <code>DDR3_1600_8x8</code> DRAM, but can have more memory types. Notably, aarch64 has as shown on RealView.py <code>VExpress_GEM5_Base</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x00000000-0x03ffffff: (  0     -  64 MiB) Boot memory (CS0)
0x04000000-0x07ffffff: ( 64 MiB - 128 MiB) Reserved
0x08000000-0x0bffffff: (128 MiB - 192 MiB) NOR FLASH0 (CS0 alias)
0x0c000000-0x0fffffff: (192 MiB - 256 MiB) NOR FLASH1 (Off-chip, CS4)
0x80000000-XxXXXXXXXX: (  2 GiB -        ) DRAM</pre>
</div>
</div>
<div class="paragraph">
<p>We place the entry point of our baremetal executables right at the start of DRAM with our <a href="#baremetal-linker-script">Baremetal linker script</a>.</p>
</div>
<div class="paragraph">
<p>This can be seen indirectly with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar --arch aarch64 --emulator gem5 entry_address</pre>
</div>
</div>
<div class="paragraph">
<p>which gives 0x80000000 in decimal, or more directly with some some <a href="#gem5-tracing">gem5 tracing</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --baremetal baremetal/arch/aarch64/no_bootloader/exit.S \
  --emulator gem5 \
  --trace ExecAll,-ExecSymbol \
  --trace-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and we see that the first instruction runs at 0x80000000:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu: A0 T0 : 0x80000000</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what are the boot memory and NOR FLASH used for?</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-disk-and-network-latency"><a class="anchor" href="#gem5-disk-and-network-latency"></a><a class="link" href="#gem5-disk-and-network-latency">24.3.4. gem5 disk and network latency</a></h4>
<div class="paragraph">
<p>TODO These look promising:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--ethernet-linkspeed
--ethernet-linkdelay</pre>
</div>
</div>
<div class="paragraph">
<p>and also: <code>gem5-dist</code>: <a href="https://publish.illinois.edu/icsl-pdgem5/" class="bare">https://publish.illinois.edu/icsl-pdgem5/</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-clock-frequency"><a class="anchor" href="#gem5-clock-frequency"></a><a class="link" href="#gem5-clock-frequency">24.3.5. gem5 clock frequency</a></h4>
<div class="paragraph">
<p>As of gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 defaults to 2GHz for fs.py:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    parser.add_option("--cpu-clock", action="store", type="string",
                      default='2GHz',
                      help="Clock for blocks running at CPU speed")</pre>
</div>
</div>
<div class="paragraph">
<p>We can check that very easily by looking at the timestamps of a <a href="#gem5-execall-trace-format">Exec trace</a> of an <a href="#gem5-atomicsimplecpu">gem5 <code>AtomicSimpleCPU</code></a> without any caches:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace-insts-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which shows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
    500: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
   1000: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)
   1500: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x8, #64, #0       : IntAlu :  D=0x0000000000000040  flags=(IsInteger)
   2000: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
hello
   2500: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   3000: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
   3500: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that it runs one instruction every 500 ps which makes up 2GHz.</p>
</div>
<div class="paragraph">
<p>So if we change the frequency to say 1GHz and re-run it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace-insts-stdout \
  -- \
  --cpu-clock 1GHz \
;</pre>
</div>
</div>
<div class="paragraph">
<p>we get as expected:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
   1000: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
   2000: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)
   3000: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x8, #64, #0       : IntAlu :  D=0x0000000000000040  flags=(IsInteger)
   4000: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
hello
   5000: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   6000: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
   7000: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)</pre>
</div>
</div>
<div class="paragraph">
<p>As of gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1, but like <a href="#gem5-cache-size">gem5 cache size</a>, does not get propagated to the guest, and is not for example visible at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls /sys/devices/system/cpu/cpu0/cpufreq</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-kernel-command-line-parameters"><a class="anchor" href="#gem5-kernel-command-line-parameters"></a><a class="link" href="#gem5-kernel-command-line-parameters">24.4. gem5 kernel command line parameters</a></h3>
<div class="paragraph">
<p>Analogous <a href="#kernel-command-line-parameters">to QEMU</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --kernel-cli 'init=/lkmc/linux/poweroff.out' --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Internals: when we give <code>--command-line=</code> to gem5, it overrides default command lines, including some mandatory ones which are required to boot properly.</p>
</div>
<div class="paragraph">
<p>Our run script hardcodes the require options in the default <code>--command-line</code> and appends extra options given by <code>-e</code>.</p>
</div>
<div class="paragraph">
<p>To find the default options in the first place, we removed <code>--command-line</code> and ran:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>and then looked at the line of the Linux kernel that starts with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Kernel command line:</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-gdb"><a class="anchor" href="#gem5-gdb"></a><a class="link" href="#gem5-gdb">24.5. gem5 GDB step debug</a></h3>
<div class="sect3">
<h4 id="gem5-gdb-step-debug-kernel"><a class="anchor" href="#gem5-gdb-step-debug-kernel"></a><a class="link" href="#gem5-gdb-step-debug-kernel">24.5.1. gem5 GDB step debug kernel</a></h4>
<div class="paragraph">
<p>Analogous <a href="#gdb">to QEMU</a>, on the first shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>On the second shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>On a third shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-shell</pre>
</div>
</div>
<div class="paragraph">
<p>When you want to break, just do a <code>Ctrl-C</code> on GDB shell, and then <code>continue</code>.</p>
</div>
<div class="paragraph">
<p>And we now see the boot messages, and then get a shell. Now try the <code>./count.sh</code> procedure described for QEMU at: <a href="#gdb-step-debug-kernel-post-boot">Section 3.2, &#8220;GDB step debug kernel post-boot&#8221;</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-gdb-step-debug-userland-process"><a class="anchor" href="#gem5-gdb-step-debug-userland-process"></a><a class="link" href="#gem5-gdb-step-debug-userland-process">24.5.2. gem5 GDB step debug userland process</a></h4>
<div class="paragraph">
<p>We are unable to use <code>gdbserver</code> because of networking as mentioned at: <a href="#gem5-host-to-guest-networking">Section 15.3.1.3, &#8220;gem5 host to guest networking&#8221;</a></p>
</div>
<div class="paragraph">
<p>The alternative is to do as in <a href="#gdb-step-debug-userland-processes">GDB step debug userland processes</a>.</p>
</div>
<div class="paragraph">
<p>Next, follow the exact same steps explained at <a href="#gdb-step-debug-userland-non-init-without-gdb-wait">GDB step debug userland non-init without --gdb-wait</a>, but passing <code>--emulator gem5</code> to every command as usual.</p>
</div>
<div class="paragraph">
<p>But then TODO (I&#8217;ll still go crazy one of those days): for <code>arm</code>, while debugging <code>./linux/myinsmod.out hello.ko</code>, after then line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>23     if (argc &lt; 3) {
24         params = "";</pre>
</div>
</div>
<div class="paragraph">
<p>I press <code>n</code>, it just runs the program until the end, instead of stopping on the next line of execution. The module does get inserted normally.</p>
</div>
<div class="paragraph">
<p>TODO:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm --emulator gem5 --userland gem5-1.0/gem5/util/m5/m5 main</pre>
</div>
</div>
<div class="paragraph">
<p>breaks when <code>m5</code> is run on guest, but does not show the source code.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-gdb-step-debug-secondary-cores"><a class="anchor" href="#gem5-gdb-step-debug-secondary-cores"></a><a class="link" href="#gem5-gdb-step-debug-secondary-cores">24.5.3. gem5 GDB step debug secondary cores</a></h4>
<div class="paragraph">
<p>gem5&#8217;s secondary core GDB setup is a hack and spawns one gdbserver for each core in separate ports, e.g. 7000, 7001, etc.</p>
</div>
<div class="paragraph">
<p>Partly because of this, it is basically unusable/very hard to use, because you can&#8217;t attach to a core that is stopped either because it hasn&#8217;t been initialized, or if you are already currently debugging another core.</p>
</div>
<div class="paragraph">
<p>This affects both full system and <a href="#gdb-step-debug-multicore-userland">userland</a>, and is described in more detail at: <a href="https://gem5.atlassian.net/browse/GEM5-626" class="bare">https://gem5.atlassian.net/browse/GEM5-626</a></p>
</div>
<div class="paragraph">
<p>In LKMC 0a3ce2f41f12024930bcdc74ff646b66dfc46999, we can easily test attaching to another core by passing <code>--run-id</code>, e.g. to connect to the second core we can use <code>--run-id 1</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch aarch64 --emulator gem5 --userland userland/gcc/busy_loop.c --run-id 1</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-checkpoint"><a class="anchor" href="#gem5-checkpoint"></a><a class="link" href="#gem5-checkpoint">24.6. gem5 checkpoint</a></h3>
<div class="paragraph">
<p>Analogous to QEMU&#8217;s <a href="#snapshot">Snapshot</a>, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before <code>init</code> is done.</p>
</div>
<div class="paragraph">
<p>Documentation: <a href="http://gem5.org/Checkpoints" class="bare">http://gem5.org/Checkpoints</a></p>
</div>
<div class="paragraph">
<p>To see it in action try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>In the guest, wait for the boot to end and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 checkpoint</pre>
</div>
</div>
<div class="paragraph">
<p>where <a href="#gem5-m5-executable">gem5 m5 executable</a> is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest.</p>
</div>
<div class="paragraph">
<p>To restore the checkpoint, kill the VM and run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 --gem5-restore 1</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>--gem5-restore</code> option restores the checkpoint that was created most recently.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s create a second checkpoint to see how it works, in guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>date &gt;f
m5 checkpoint</pre>
</div>
</div>
<div class="paragraph">
<p>Kill the VM, and try it out:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 --gem5-restore 1</pre>
</div>
</div>
<div class="paragraph">
<p>Here we use <code>--gem5-restore 1</code> again, since the second snapshot we took is now the most recent one</p>
</div>
<div class="paragraph">
<p>Now in the guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat f</pre>
</div>
</div>
<div class="paragraph">
<p>contains the <code>date</code>. The file <code>f</code> wouldn&#8217;t exist had we used the first checkpoint with <code>--gem5-restore 2</code>, which is the second most recent snapshot taken.</p>
</div>
<div class="paragraph">
<p>If you automate things with <a href="#kernel-command-line-parameters">Kernel command line parameters</a> as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --eval 'm5 checkpoint;m5 resetstats;dhrystone 1000;m5 exit' --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Then there is no need to pass the kernel command line again to gem5 for replay:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 --gem5-restore 1</pre>
</div>
</div>
<div class="paragraph">
<p>since boot has already happened, and the parameters are already in the RAM of the snapshot.</p>
</div>
<div class="sect3">
<h4 id="gem5-checkpoint-userland-minimal-example"><a class="anchor" href="#gem5-checkpoint-userland-minimal-example"></a><a class="link" href="#gem5-checkpoint-userland-minimal-example">24.6.1. gem5 checkpoint userland minimal example</a></h4>
<div class="paragraph">
<p>In order to debug checkpoint restore bugs, this minimal setup using <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/gem5_checkpoint.S">userland/freestanding/gem5_checkpoint.S</a> can be handy:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --arch aarch64 --static
./run --arch aarch64 --emulator gem5 --static --userland userland/freestanding/gem5_checkpoint.S --trace-insts-stdout
./run --arch aarch64 --emulator gem5 --static --userland userland/freestanding/gem5_checkpoint.S --trace-insts-stdout --gem5-restore 1
./run --arch aarch64 --emulator gem5 --static --userland userland/freestanding/gem5_checkpoint.S --trace-insts-stdout --gem5-restore 1 -- --cpu-type=DerivO3CPU --restore-with-cpu=DerivO3CPU --caches</pre>
</div>
</div>
<div class="paragraph">
<p>On the initial run, we see that all instructions are executed and the checkpoint is taken:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
    500: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   movz   x1, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   1000: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   m5checkpoint             : IntAlu :   flags=(IsInteger|IsNonSpeculative|IsUnverifiable)
Writing checkpoint
warn: Checkpoints for file descriptors currently do not work.
info: Entering event queue @ 1000.  Starting simulation...
   1500: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   2000: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   m5exit                   : No_OpClass :   flags=(IsInteger|IsNonSpeculative)
Exiting @ tick 2000 because m5_exit instruction encountered</pre>
</div>
</div>
<div class="paragraph">
<p>Then, on the first restore run, the checkpoint is restored, and only instructions after the checkpoint are executed:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: Entering event queue @ 1000.  Starting simulation...
   1500: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   2000: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   m5exit                   : No_OpClass :   flags=(IsInteger|IsNonSpeculative)
Exiting @ tick 2000 because m5_exit instruction encountered</pre>
</div>
</div>
<div class="paragraph">
<p>and a similar thing happens for the <a href="#gem5-restore-checkpoint-with-a-different-cpu">restore with a different CPU type</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: Entering event queue @ 1000.  Starting simulation...
  79000: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  FetchSeq=1  CPSeq=1  flags=(IsInteger)
Exiting @ tick 84500 because m5_exit instruction encountered</pre>
</div>
</div>
<div class="paragraph">
<p>Here we don&#8217;t see the last <code>m5 exit</code> instruction on the log, but it must just be something to do with the O3 logging.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-checkpoint-internals"><a class="anchor" href="#gem5-checkpoint-internals"></a><a class="link" href="#gem5-checkpoint-internals">24.6.2. gem5 checkpoint internals</a></h4>
<div class="paragraph">
<p>A quick way to get a <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a> or full system checkpoint to observe is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --baremetal userland/freestanding/gem5_checkpoint.S --trace-insts-stdout
./run --arch aarch64 --emulator gem5 --userland userland/freestanding/gem5_checkpoint.S --trace-insts-stdout</pre>
</div>
</div>
<div class="paragraph">
<p>Checkpoints are stored inside the <a href="#m5out-directory">m5out directory</a> at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>"$(./getvar --emulator gem5 m5out_dir)/cpt.&lt;checkpoint-time&gt;"</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>&lt;checkpoint-time&gt;</code> is the cycle number at which the checkpoint was taken.</p>
</div>
<div class="paragraph">
<p><code>fs.py</code> exposes the <code>-r N</code> flag to restore checkpoints, which N-th checkpoint with the largest <code>&lt;checkpoint-time&gt;</code>: <a href="https://github.com/gem5/gem5/blob/e02ec0c24d56bce4a0d8636a340e15cd223d1930/configs/common/Simulation.py#L118" class="bare">https://github.com/gem5/gem5/blob/e02ec0c24d56bce4a0d8636a340e15cd223d1930/configs/common/Simulation.py#L118</a></p>
</div>
<div class="paragraph">
<p>However, that interface is bad because if you had taken previous checkpoints, you have no idea what <code>N</code> to use, unless you memorize which checkpoint was taken at which cycle.</p>
</div>
<div class="paragraph">
<p>Therefore, just use our superior <code>--gem5-restore</code> flag, which uses directory timestamps to determine which checkpoint you created most recently.</p>
</div>
<div class="paragraph">
<p>The <code>-r N</code> integer value is just pure <code>fs.py</code> sugar, the backend at <code>m5.instantiate</code> just takes the actual tracepoint directory path as input.</p>
</div>
<div class="paragraph">
<p>The file <code>m5out/cpt.1000/m5.cpt</code> contains almost everything in the checkpoint except memory.</p>
</div>
<div class="paragraph">
<p>It is a <a href="https://docs.python.org/3/library/configparser.html">Python configparser compatible file</a> with a section structure that matches the <a href="#gem5-python-c-interaction">SimObject</a> tree e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.itb.walker.power_state]
currState=0
prvEvalTick=0</pre>
</div>
</div>
<div class="paragraph">
<p>When a checkpoint is taken, each <code>SimObject</code> calls its overridden <code>serialize</code> method to generate the checkpoint, and when loading, <code>unserialize</code> is called.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-restore-new-script"><a class="anchor" href="#gem5-restore-new-script"></a><a class="link" href="#gem5-restore-new-script">24.6.3. gem5 checkpoint restore and run a different script</a></h4>
<div class="paragraph">
<p>You want to automate running several tests from a single pristine post-boot state.</p>
</div>
<div class="paragraph">
<p>The problem is that boot takes forever, and after the checkpoint, the memory and disk states are fixed, so you can&#8217;t for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>hack up an existing rc script, since the disk is fixed</p>
</li>
<li>
<p>inject new kernel boot command line options, since those have already been put into memory by the bootloader</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>There is however a few loopholes, <a href="#m5-readfile">m5 readfile</a> being the simplest, as it reads whatever is present on the host.</p>
</div>
<div class="paragraph">
<p>So we can do it like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Boot, checkpoint and exit.
printf 'echo "setup run";m5 exit' &gt; "$(./getvar gem5_readfile_file)"
./run --emulator gem5 --eval 'm5 checkpoint;m5 readfile &gt; /tmp/gem5.sh &amp;&amp; sh /tmp/gem5.sh'

# Restore and run the first benchmark.
printf 'echo "first benchmark";m5 exit' &gt; "$(./getvar gem5_readfile_file)"
./run --emulator gem5 --gem5-restore 1

# Restore and run the second benchmark.
printf 'echo "second benchmark";m5 exit' &gt; "$(./getvar gem5_readfile_file)"
./run --emulator gem5 --gem5-restore 1

# If something weird happened, create an interactive shell to examine the system.
printf 'sh' &gt; "$(./getvar gem5_readfile_file)"
./run --emulator gem5 --gem5-restore 1</pre>
</div>
</div>
<div class="paragraph">
<p>Since this is such a common setup, we provide the following helpers for this operation:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>./run --gem5-readfile</code> is a convenient way to set the <code>m5 readfile</code> file contents from a string in the command line, e.g.:</p>
<div class="literalblock">
<div class="content">
<pre># Boot, checkpoint and exit.
./run --emulator gem5 --eval './gem5.sh' --gem5-readfile 'echo "setup run"'

# Restore and run the first benchmark.
./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "first benchmark"'

# Restore and run the second benchmark.
./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "second benchmark"'</pre>
</div>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/gem5.sh">rootfs_overlay/lkmc/gem5.sh</a>. This script is analogous to gem5&#8217;s in-tree <a href="https://github.com/gem5/gem5/blob/2b4b94d0556c2d03172ebff63f7fc502c3c26ff8/configs/boot/hack_back_ckpt.rcS">hack_back_ckpt.rcS</a>, but with less noise.</p>
<div class="paragraph">
<p>Usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Boot, checkpoint and exit.
./run --emulator gem5 --eval './gem5.sh' --gem5-readfile 'echo "setup run"'

# Restore and run the first benchmark.
./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "first benchmark"'

# Restore and run the second benchmark.
./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "second benchmark"'</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Their usage is also exemplified at <a href="#gem5-run-benchmark">gem5 run benchmark</a>.</p>
</div>
<div class="paragraph">
<p>If you forgot to use an appropriate <code>--eval</code> for your boot and the simulation is already running, <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/gem5.sh">rootfs_overlay/lkmc/gem5.sh</a> can be used directly from an interactive guest shell.</p>
</div>
<div class="paragraph">
<p>First we reset the readfile to something that runs quickly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf 'echo "first benchmark"' &gt; "$(./getvar gem5_readfile_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>and then in the guest, take a checkpoint and exit with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5.sh</pre>
</div>
</div>
<div class="paragraph">
<p>Now the guest is in a state where readfile will be executed automatically without interactive intervention:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "first benchmark"'
./run --emulator gem5 --gem5-restore 1 --gem5-readfile 'echo "second benchmark"'</pre>
</div>
</div>
<div class="paragraph">
<p>Other loophole possibilities to execute different benchmarks non-interactively include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#9p">9P</a></p>
</li>
<li>
<p><a href="#secondary-disk">Secondary disk</a></p>
</li>
<li>
<p><code>expect</code> as mentioned at: <a href="https://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts" class="bare">https://stackoverflow.com/questions/7013137/automating-telnet-session-using-bash-scripts</a></p>
<div class="literalblock">
<div class="content">
<pre>#!/usr/bin/expect
spawn telnet localhost 3456
expect "# $"
send "pwd\r"
send "ls /\r"
send "m5 exit\r"
expect eof</pre>
</div>
</div>
<div class="paragraph">
<p>This is ugly however as it is not deterministic.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p><a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15233.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15233.html</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-restore-checkpoint-with-a-different-cpu"><a class="anchor" href="#gem5-restore-checkpoint-with-a-different-cpu"></a><a class="link" href="#gem5-restore-checkpoint-with-a-different-cpu">24.6.4. gem5 restore checkpoint with a different CPU</a></h4>
<div class="paragraph">
<p>gem5 can switch to a different CPU model when restoring a checkpoint.</p>
</div>
<div class="paragraph">
<p>A common combo is to boot Linux with a fast CPU, make a checkpoint and then replay the benchmark of interest with a slower CPU.</p>
</div>
<div class="paragraph">
<p>This can be observed interactively in full system with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Then in the guest terminal after boot ends:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sh -c 'm5 checkpoint;sh'
m5 exit</pre>
</div>
</div>
<div class="paragraph">
<p>And then restore the checkpoint with a different slower CPU:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 --gem5-restore 1 -- --caches --cpu-type=DerivO3CPU</pre>
</div>
</div>
<div class="paragraph">
<p>And now you will notice that everything happens much slower in the guest terminal!</p>
</div>
<div class="paragraph">
<p>One even more direct and minimal way to observe this is with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/gem5_checkpoint.S">userland/freestanding/gem5_checkpoint.S</a> which was mentioned at <a href="#gem5-checkpoint-userland-minimal-example">gem5 checkpoint userland minimal example</a> plus some logging:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --static \
  --trace ExecAll,FmtFlag,O3CPU,SimpleCPU \
  --userland userland/freestanding/gem5_checkpoint.S \
;
cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"
./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-restore 1 \
  --static \
  --trace ExecAll,FmtFlag,O3CPU,SimpleCPU \
  --userland userland/freestanding/gem5_checkpoint.S \
  -- \
  --caches \
  --cpu-type DerivO3CPU \
  --restore-with-cpu DerivO3CPU \
;
cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>At gem5 2235168b72537535d74c645a70a85479801e0651, the first run does everything in <a href="#gem5-atomicsimplecpu">AtomicSimpleCPU</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>...
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1f92 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e40 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e30 WriteReq
      0: SimpleCPU: system.cpu: Tick
      0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
    500: SimpleCPU: system.cpu: Tick
    500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   movz   x1, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   1000: SimpleCPU: system.cpu: Tick
   1000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   m5checkpoint             : IntAlu :   flags=(IsInteger|IsNonSpeculative|IsUnverifiable)
   1000: SimpleCPU: system.cpu: Resume
   1500: SimpleCPU: system.cpu: Tick
   1500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   2000: SimpleCPU: system.cpu: Tick
   2000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   m5exit                   : No_OpClass :   flags=(IsInteger|IsNonSpeculative)</pre>
</div>
</div>
<div class="paragraph">
<p>and after restore we see as expected a single <code>ExecEnable</code> instruction executed amidst <code>O3CPU</code> noise:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FullO3CPU: Ticking main, FullO3CPU.
  79000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  FetchSeq=1  CPSeq=1  flags=(IsInteger)
  82500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400084=&gt;0x400088).(0=&gt;1) [sn:1]
  82500: O3CPU: system.cpu: Removing instruction, [tid:0] [sn:1] PC (0x400084=&gt;0x400088).(0=&gt;1)
  82500: O3CPU: system.cpu: Scheduling next tick!
  83000: O3CPU: system.cpu:</pre>
</div>
</div>
<div class="paragraph">
<p>which is the <code>movz</code> after the checkpoint. The final <code>m5exit</code> does not appear due to DerivO3CPU logging insanity.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/60876259/which-system-characteristics-such-as-number-of-cores-of-cache-configurations-can" class="bare">https://stackoverflow.com/questions/60876259/which-system-characteristics-such-as-number-of-cores-of-cache-configurations-can</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/49011096/how-to-switch-cpu-models-in-gem5-after-restoring-a-checkpoint-and-then-observe-t" class="bare">https://stackoverflow.com/questions/49011096/how-to-switch-cpu-models-in-gem5-after-restoring-a-checkpoint-and-then-observe-t</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="gem5-fast-forward"><a class="anchor" href="#gem5-fast-forward"></a><a class="link" href="#gem5-fast-forward">24.6.4.1. gem5 fast forward</a></h5>
<div class="paragraph">
<p>Besides switching CPUs after a checkpoint restore, fs.py also has the <code>--fast-forward</code> option to automatically run the script from the start on a less detailed CPU, and switch to a more detailed CPU at a given tick.</p>
</div>
<div class="paragraph">
<p>This is generally useless compared to checkpoint restoring because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>checkpoint restore allows to run multiple contents after the restore, and restoring to multiple different system states, which you almost always want to do</p>
</li>
<li>
<p>we generally don&#8217;t know the exact tick at which the region of interest will start, especially as the binaries change. It is much easier to just instrument the content with a checkoint <a href="#m5ops">m5op</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>But let&#8217;s give it a try anyway with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/gem5_checkpoint.S">userland/freestanding/gem5_checkpoint.S</a> which was mentioned at <a href="#gem5-checkpoint-userland-minimal-example">gem5 checkpoint userland minimal example</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --static \
  --trace ExecAll,FmtFlag,O3CPU,SimpleCPU \
  --userland userland/freestanding/gem5_checkpoint.S \
  -- \
  --caches
  --cpu-type DerivO3CPU \
  --fast-forward 1000 \
;
cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>At gem5 2235168b72537535d74c645a70a85479801e0651 we see something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: O3CPU: system.switch_cpus: Creating O3CPU object.
      0: O3CPU: system.switch_cpus: Workload[0] process is 0      0: SimpleCPU: system.cpu: ActivateContext 0
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x40 WriteReq
...

      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1f92 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e40 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e30 WriteReq
      0: SimpleCPU: system.cpu: Tick
      0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
    500: SimpleCPU: system.cpu: Tick
    500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   movz   x1, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   1000: SimpleCPU: system.cpu: Tick
   1000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   m5checkpoint             : IntAlu :   flags=(IsInteger|IsNonSpeculative|IsUnverifiable)
   1000: O3CPU: system.switch_cpus: [tid:0] Calling activate thread.
   1000: O3CPU: system.switch_cpus: [tid:0] Adding to active threads list
   1500: O3CPU: system.switch_cpus:

FullO3CPU: Ticking main, FullO3CPU.
   1500: O3CPU: system.switch_cpus: Scheduling next tick!
   2000: O3CPU: system.switch_cpus:

FullO3CPU: Ticking main, FullO3CPU.
   2000: O3CPU: system.switch_cpus: Scheduling next tick!
   2500: O3CPU: system.switch_cpus:

...

FullO3CPU: Ticking main, FullO3CPU.
  44500: ExecEnable: system.switch_cpus: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x00000000000
  48000: O3CPU: system.switch_cpus: Removing committed instruction [tid:0] PC (0x400084=&gt;0x400088).(0=&gt;1) [sn:1]
  48000: O3CPU: system.switch_cpus: Removing instruction, [tid:0] [sn:1] PC (0x400084=&gt;0x400088).(0=&gt;1)
  48000: O3CPU: system.switch_cpus: Scheduling next tick!
  48500: O3CPU: system.switch_cpus:

...</pre>
</div>
</div>
<div class="paragraph">
<p>We can also compare that to the same log but without <code>--fast-forward</code> and other CPU switch options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e40 WriteReq
      0: SimpleCPU: system.cpu.dcache_port: received snoop pkt for addr:0x1e30 WriteReq
      0: SimpleCPU: system.cpu: Tick
      0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
    500: SimpleCPU: system.cpu: Tick
    500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   movz   x1, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   1000: SimpleCPU: system.cpu: Tick
   1000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   m5checkpoint             : IntAlu :   flags=(IsInteger|IsNonSpeculative|IsUnverifiable)
   1000: SimpleCPU: system.cpu: Resume
   1500: SimpleCPU: system.cpu: Tick
   1500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   2000: SimpleCPU: system.cpu: Tick
   2000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   m5exit                   : No_OpClass :   flags=(IsInteger|IsNonSpeculative)</pre>
</div>
</div>
<div class="paragraph">
<p>Therefore, it is clear that what we wanted happen:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>up until the tick 1000, <code>SimpleCPU</code> was ticking</p>
</li>
<li>
<p>after tick 1000, cpu <code>O3CPU</code> started ticking</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://cs.stackexchange.com/questions/69511/what-does-fast-forwarding-mean-in-the-context-of-cpu-simulation" class="bare">https://cs.stackexchange.com/questions/69511/what-does-fast-forwarding-mean-in-the-context-of-cpu-simulation</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-checkpoint-upgrader"><a class="anchor" href="#gem5-checkpoint-upgrader"></a><a class="link" href="#gem5-checkpoint-upgrader">24.6.5. gem5 checkpoint upgrader</a></h4>
<div class="paragraph">
<p>The in-tree <code>util/cpt_upgrader.py</code> is a tool to upgrade checkpoints taken from an older version of gem5 to be compatible with the newest version, so you can update gem5 without having to re-run the simulation that generated the checkpoints.</p>
</div>
<div class="paragraph">
<p>For example, whenever a <a href="#arm-system-register-instructions">system register is added in ARMv8</a>, old checkpoints break unless upgraded.</p>
</div>
<div class="paragraph">
<p>Unfortunately, since the process is not very automated (automatable?), and requires manually patching the upgrader every time a new breaking change is done, the upgrader tends to break soon if you try to move many versions of gem5 ahead as of 2020. This is evidenced in bug reports such as this one: <a href="https://gem5.atlassian.net/browse/GEM5-472" class="bare">https://gem5.atlassian.net/browse/GEM5-472</a></p>
</div>
<div class="paragraph">
<p>The script can be used as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>util/cpt_upgrader.py m5out/cpt.1000/m5.cpt</pre>
</div>
</div>
<div class="paragraph">
<p>This updates the <code>m5.cpt</code> file in-place, and a <code>m5out/cpt.1000/m5.cpt.bak</code> is generated as a backup of the old file.</p>
</div>
<div class="paragraph">
<p>The upgrader determines which upgrades are needed by checking the <code>version_tags</code> entry of the checkpoint:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[Globals]
version_tags=arm-ccregs arm-contextidr-el2 arm-gem5-gic-ext ...</pre>
</div>
</div>
<div class="paragraph">
<p>Each of those tags corresponds to a Python file under <code>util/cpt_upgraders/</code> e.g. <code>util/cpt_upgraders/arm-ccregs.py</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="pass-extra-options-to-gem5"><a class="anchor" href="#pass-extra-options-to-gem5"></a><a class="link" href="#pass-extra-options-to-gem5">24.7. Pass extra options to gem5</a></h3>
<div class="paragraph">
<p>Remember that in the gem5 command line, we can either pass options to the script being run as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/X86/gem5.opt configs/examples/fs.py --some-option</pre>
</div>
</div>
<div class="paragraph">
<p>or to the gem5 executable itself:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/X86/gem5.opt --some-option configs/examples/fs.py</pre>
</div>
</div>
<div class="paragraph">
<p>Pass options to the script in our setup use:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>get help:</p>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 -- -h</pre>
</div>
</div>
</li>
<li>
<p>boot with the more detailed and slow <code>HPI</code> CPU model:</p>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --emulator gem5 -- --caches --cpu-type=HPI</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>To pass options to the <code>gem5</code> executable we expose the <code>--gem5-exe-args</code> option:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>get help:</p>
<div class="literalblock">
<div class="content">
<pre>./run --gem5-exe-args='-h' --emulator gem5</pre>
</div>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="m5ops"><a class="anchor" href="#m5ops"></a><a class="link" href="#m5ops">24.8. m5ops</a></h3>
<div class="paragraph">
<p>m5ops are magic instructions which lead gem5 to do magic things, like quitting or dumping stats.</p>
</div>
<div class="paragraph">
<p>Documentation: <a href="http://gem5.org/M5ops" class="bare">http://gem5.org/M5ops</a></p>
</div>
<div class="paragraph">
<p>There are two main ways to use m5ops:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-m5-executable">gem5 m5 executable</a></p>
</li>
<li>
<p><a href="#m5ops-instructions">m5ops instructions</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><code>m5</code> is convenient if you only want to take snapshots before or after the benchmark, without altering its source code. It uses the <a href="#m5ops-instructions">m5ops instructions</a> as its backend.</p>
</div>
<div class="paragraph">
<p><code>m5</code> cannot should / should not be used however:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>in bare metal setups</p>
</li>
<li>
<p>when you want to call the instructions from inside interest points of your benchmark. Otherwise you add the syscall overhead to the benchmark, which is more intrusive and might affect results.</p>
<div class="paragraph">
<p>Why not just hardcode some <a href="#m5ops-instructions">m5ops instructions</a> as in our example instead, since you are going to modify the source of the benchmark anyway?</p>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gem5-m5-executable"><a class="anchor" href="#gem5-m5-executable"></a><a class="link" href="#gem5-m5-executable">24.8.1. gem5 m5 executable</a></h4>
<div class="paragraph">
<p><code>m5</code> is a guest command line utility that is installed and run on the guest, that serves as a CLI front-end for the <a href="#m5ops">m5ops</a></p>
</div>
<div class="paragraph">
<p>Its source is present in the gem5 tree: <a href="https://github.com/gem5/gem5/blob/6925bf55005c118dc2580ba83e0fa10b31839ef9/util/m5/m5.c" class="bare">https://github.com/gem5/gem5/blob/6925bf55005c118dc2580ba83e0fa10b31839ef9/util/m5/m5.c</a></p>
</div>
<div class="paragraph">
<p>It is possible to guess what most tools do from the corresponding <a href="#m5ops">m5ops</a>, but let&#8217;s at least document the less obvious ones here.</p>
</div>
<div class="paragraph">
<p>In LKMC we build <code>m5</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-m5 --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>m5</code> executable can be run on <a href="#user-mode-simulation">User mode simulation</a> as normal with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --userland "$(./getvar --arch aarch64 out_rootfs_overlay_bin_dir)/m5" --cli-args dumpstats</pre>
</div>
</div>
<div class="paragraph">
<p>This can be a good test <a href="#m5ops">m5ops</a> since it executes very quickly.</p>
</div>
<div class="sect4">
<h5 id="m5-exit"><a class="anchor" href="#m5-exit"></a><a class="link" href="#m5-exit">24.8.1.1. m5 exit</a></h5>
<div class="paragraph">
<p>End the simulation.</p>
</div>
<div class="paragraph">
<p>Sane Python scripts will exit gem5 with status 0, which is what <code>fs.py</code> does.</p>
</div>
</div>
<div class="sect4">
<h5 id="m5-dumpstats"><a class="anchor" href="#m5-dumpstats"></a><a class="link" href="#m5-dumpstats">24.8.1.2. m5 dumpstats</a></h5>
<div class="paragraph">
<p>Makes gem5 dump one more statistics entry to the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="m5-fail"><a class="anchor" href="#m5-fail"></a><a class="link" href="#m5-fail">24.8.1.3. m5 fail</a></h5>
<div class="paragraph">
<p>End the simulation with a failure exit event:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 fail 1</pre>
</div>
</div>
<div class="paragraph">
<p>Sane Python scripts would use that as the exit status of gem5, which would be useful for testing purposes, but <code>fs.py</code> at 200281b08ca21f0d2678e23063f088960d3c0819 just prints an error message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Simulated exit code not 0! Exit code is 1</pre>
</div>
</div>
<div class="paragraph">
<p>and exits with status 0.</p>
</div>
<div class="paragraph">
<p>We then parse that string ourselves in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> and exit with the correct status&#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>TODO: it used to be like that, but it actually got changed to just print the message. Why? <a href="https://gem5-review.googlesource.com/c/public/gem5/+/4880" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/4880</a></p>
</div>
<div class="paragraph">
<p><code>m5 fail</code> is just a superset of <code>m5 exit</code>, which is just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 fail 0</pre>
</div>
</div>
<div class="paragraph">
<p>as can be seen from the source: <a href="https://github.com/gem5/gem5/blob/50a57c0376c02c912a978c4443dd58caebe0f173/src/sim/pseudo_inst.cc#L303" class="bare">https://github.com/gem5/gem5/blob/50a57c0376c02c912a978c4443dd58caebe0f173/src/sim/pseudo_inst.cc#L303</a></p>
</div>
</div>
<div class="sect4">
<h5 id="m5-writefile"><a class="anchor" href="#m5-writefile"></a><a class="link" href="#m5-writefile">24.8.1.4. m5 writefile</a></h5>
<div class="paragraph">
<p>Send a guest file to the host. <a href="#9p">9P</a> is a more advanced alternative.</p>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo mycontent &gt; myfileguest
m5 writefile myfileguest myfilehost</pre>
</div>
</div>
<div class="paragraph">
<p>Host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar --arch aarch64 --emulator gem5 m5out_dir)/myfilehost"</pre>
</div>
</div>
<div class="paragraph">
<p>Does not work for subdirectories, gem5 crashes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 writefile myfileguest mydirhost/myfilehost</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="m5-readfile"><a class="anchor" href="#m5-readfile"></a><a class="link" href="#m5-readfile">24.8.1.5. m5 readfile</a></h5>
<div class="paragraph">
<p>Read a host file pointed to by the <code>fs.py --script</code> option to stdout.</p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/49516399/how-to-use-m5-readfile-and-m5-execfile-in-gem5/49538051#49538051" class="bare">https://stackoverflow.com/questions/49516399/how-to-use-m5-readfile-and-m5-execfile-in-gem5/49538051#49538051</a></p>
</div>
<div class="paragraph">
<p>Host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>date &gt; "$(./getvar gem5_readfile_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 readfile</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: date shows on guest.</p>
</div>
</div>
<div class="sect4">
<h5 id="m5-initparam"><a class="anchor" href="#m5-initparam"></a><a class="link" href="#m5-initparam">24.8.1.6. m5 initparam</a></h5>
<div class="paragraph">
<p>Ermm, just another <a href="#m5-readfile">m5 readfile</a> that only takes integers and only from CLI options? Is this software so redundant?</p>
</div>
<div class="paragraph">
<p>Host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --gem5-restore 1 -- --initparam 13
./run --emulator gem5 --gem5-restore 1 -- --initparam 42</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 initparm</pre>
</div>
</div>
<div class="paragraph">
<p>Outputs the given paramter.</p>
</div>
</div>
<div class="sect4">
<h5 id="m5-execfile"><a class="anchor" href="#m5-execfile"></a><a class="link" href="#m5-execfile">24.8.1.7. m5 execfile</a></h5>
<div class="paragraph">
<p>Trivial combination of <code>m5 readfile</code> + execute the script.</p>
</div>
<div class="paragraph">
<p>Host:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '#!/bin/sh
echo asdf
' &gt; "$(./getvar gem5_readfile_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>Guest:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>touch /tmp/execfile
chmod +x /tmp/execfile
m5 execfile</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>adsf</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="m5ops-instructions"><a class="anchor" href="#m5ops-instructions"></a><a class="link" href="#m5ops-instructions">24.8.2. m5ops instructions</a></h4>
<div class="paragraph">
<p>There are few different possible instructions that can be used to implement identical m5ops:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>magic instructions reserved in the encoding space</p>
</li>
<li>
<p>magic addresses: <a href="#m5ops-magic-addresses">m5ops magic addresses</a></p>
</li>
<li>
<p>unused <a href="#semihosting">Semihosting</a> addresses space on ARM platforms</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>All of those those methods are exposed through the <a href="#gem5-m5-executable">gem5 m5 executable</a> in-tree executable. You can select which method to use when calling the executable, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5 exit
# Same as the above.
m5 --inst exit
# The address is mandatory if not configured at build time.
m5 --addr 0x10010000 exit
m5 --semi exit</pre>
</div>
</div>
<div class="paragraph">
<p>To make things simpler to understand, you can play around with our own minimized educational <code>m5</code> subset:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/m5ops.c">userland/c/m5ops.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/m5ops.cpp">userland/cpp/m5ops.cpp</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The instructions used by <code>./c/m5ops.out</code> are present in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/m5ops.h">lkmc/m5ops.h</a> in a very simple to understand and reuse inline assembly form.</p>
</div>
<div class="paragraph">
<p>To use that file, first rebuild <code>m5ops.out</code> with the m5ops instructions enabled and install it on the root filesystem:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland \
  --arch aarch64 \
  --force-rebuild \
  userland/c/m5ops.c \
;
./build-buildroot --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>We don&#8217;t enable <code>-DLKMC_M5OPS_ENABLE=1</code> by default on userland executables because we try to use a single image for both gem5, QEMU and <a href="#userland-setup-getting-started-natively">native</a>, and those instructions would break the latter two. We enable it in the <a href="#baremetal-setup">Baremetal setup</a> by default since we already have different images for QEMU and gem5 there.</p>
</div>
<div class="paragraph">
<p>Then, from inside <a href="#gem5-buildroot-setup">gem5 Buildroot setup</a>, test it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># checkpoint
./c/m5ops.out c

# dumpstats
./c/m5ops.out d

# exit
./c/m5ops.out e

# dump resetstats
./c/m5ops.out r</pre>
</div>
</div>
<div class="paragraph">
<p>In theory, the cleanest way to add m5ops to your benchmarks would be to do exactly what the <code>m5</code> tool does:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>include <a href="https://github.com/gem5/gem5/blob/05c4c2b566ce351ab217b2bd7035562aa7a76570/include/gem5/asm/generic/m5ops.h"><code>include/gem5/asm/generic/m5ops.h</code></a></p>
</li>
<li>
<p>link with the <code>.o</code> file under <code>util/m5</code> for the correct arch, e.g. <code>m5op_arm_A64.o</code> for aarch64.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>However, I think it is usually not worth the trouble of hacking up the build system of the benchmark to do this, and I recommend just hardcoding in a few raw instructions here and there, and managing it with version control + <code>sed</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/63488050/what-are-pseudo-instructions-for-in-gem5/63489417#63489417" class="bare">https://stackoverflow.com/questions/63488050/what-are-pseudo-instructions-for-in-gem5/63489417#63489417</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/62757008/how-to-use-m5-in-gem5-20/62759204#62759204" class="bare">https://stackoverflow.com/questions/62757008/how-to-use-m5-in-gem5-20/62759204#62759204</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/56506154/how-to-analyze-only-interest-area-in-source-code-by-using-gem5/56506419#56506419" class="bare">https://stackoverflow.com/questions/56506154/how-to-analyze-only-interest-area-in-source-code-by-using-gem5/56506419#56506419</a></p>
</li>
<li>
<p><a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15418.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15418.html</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="m5ops-magic-addresses"><a class="anchor" href="#m5ops-magic-addresses"></a><a class="link" href="#m5ops-magic-addresses">24.8.2.1. m5ops magic addresses</a></h5>
<div class="paragraph">
<p>These are magic addresses that when accessed lead to an <a href="#m5ops">m5op</a>.</p>
</div>
<div class="paragraph">
<p>The base address is given by <code>system.m5ops_base</code>, and then each m5op happens at a different address offset form that base.</p>
</div>
<div class="paragraph">
<p>If <code>system.m5ops_base</code> is 0, then the memory m5ops are disabled.</p>
</div>
<div class="paragraph">
<p>Note that the address is physical, and therefore when running in full system on top of the Linux kernel, you must first map a virtual to physical address with <code>/dev/mem</code> as mentioned at: <a href="#userland-physical-address-experiments">Userland physical address experiments</a>.</p>
</div>
<div class="paragraph">
<p>One advantage of this method is that it can work with <a href="#gem5-kvm">gem5 KVM</a>, whereas the magic instructions don&#8217;t, since the host cannot handle them and it is hard to hook into that.</p>
</div>
<div class="paragraph">
<p>A <a href="#baremetal">Baremetal</a> example of that can be found at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/no_bootloader/m5_exit_addr.S">baremetal/arch/aarch64/no_bootloader/m5_exit_addr.S</a>.</p>
</div>
<div class="paragraph">
<p>As of gem5 0d5a80cb469f515b95e03f23ddaf70c9fd2ecbf2, <code>fs.py --baremetal</code> disables the memory m5ops however for some reason, therefore you should run that program as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/no_bootloader/m5_exit_addr.S --emulator gem5 --trace-insts-stdout -- --param 'system.m5ops_base=0x10010000'</pre>
</div>
</div>
<div class="paragraph">
<p>TODO failing with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: Entering event queue @ 0.  Starting simulation...
fatal: Unable to find destination for [0x10012100:0x10012108] on system.iobus</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="m5ops-instructions-interface"><a class="anchor" href="#m5ops-instructions-interface"></a><a class="link" href="#m5ops-instructions-interface">24.8.2.2. m5ops instructions interface</a></h5>
<div class="paragraph">
<p>Let&#8217;s study how the <a href="#gem5-m5-executable">gem5 m5 executable</a> uses them:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/gem5/gem5/blob/05c4c2b566ce351ab217b2bd7035562aa7a76570/include/gem5/asm/generic/m5ops.h"><code>include/gem5/asm/generic/m5ops.h</code></a>: defines the magic constants that represent the instructions</p>
</li>
<li>
<p><a href="https://github.com/gem5/gem5/blob/05c4c2b566ce351ab217b2bd7035562aa7a76570/util/m5/m5op_arm_A64.S"><code>util/m5/m5op_arm_A64.S</code></a>: use the magic constants that represent the instructions using C preprocessor magic</p>
</li>
<li>
<p><a href="https://github.com/gem5/gem5/blob/05c4c2b566ce351ab217b2bd7035562aa7a76570/util/m5/m5.c"><code>util/m5/m5.c</code></a>: the actual executable. Gets linked to <code>m5op_arm_A64.S</code> which defines a function for each m5op.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We notice that there are two different implementations for each arch:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>magic instructions, which don&#8217;t exist in the corresponding arch</p>
</li>
<li>
<p>magic memory addresses on a given page: <a href="#m5ops-magic-addresses">m5ops magic addresses</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, in aarch64 magic instructions for example, the lines:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>.macro  m5op_func, name, func, subfunc
        .globl \name
        \name:
        .long 0xff000110 | (\func &lt;&lt; 16) | (\subfunc &lt;&lt; 12)
        ret</pre>
</div>
</div>
<div class="paragraph">
<p>define a simple function function for each m5op. Here we see that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>0xff000110</code> is a base mask for the magic non-existing instruction</p>
</li>
<li>
<p><code>\func</code> and <code>\subfunc</code> are OR-applied on top of the base mask, and define m5op this is.</p>
<div class="paragraph">
<p>Those values will loop over the magic constants defined in <code>m5ops.h</code> with the deferred preprocessor idiom.</p>
</div>
<div class="paragraph">
<p>For example, <code>exit</code> is <code>0x21</code> due to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#define M5OP_EXIT               0x21</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Finally, <code>m5.c</code> calls the defined functions as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5_exit(ints[0]);</pre>
</div>
</div>
<div class="paragraph">
<p>Therefore, the runtime "argument" that gets passed to the instruction, e.g. the delay in ticks until the exit for <code>m5 exit</code>, gets passed directly through the <a href="https://en.wikipedia.org/wiki/Calling_convention#ARM_(A64)">aarch64 calling convention</a>.</p>
</div>
<div class="paragraph">
<p>Keep in mind that for all archs, <code>m5.c</code> does the calls with 64-bit integers:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>uint64_t ints[2] = {0,0};
parse_int_args(argc, argv, ints, argc);
m5_fail(ints[1], ints[0]);</pre>
</div>
</div>
<div class="paragraph">
<p>Therefore, for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>aarch64 uses <code>x0</code> for the first argument and <code>x1</code> for the second, since each is 64 bits log already</p>
</li>
<li>
<p>arm uses <code>r0</code> and <code>r1</code> for the first argument, and <code>r2</code> and <code>r3</code> for the second, since each register is only 32 bits long</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>That convention specifies that <code>x0</code> to <code>x7</code> contain the function arguments, so <code>x0</code> contains the first argument, and <code>x1</code> the second.</p>
</div>
<div class="paragraph">
<p>In our <code>m5ops</code> example, we just hardcode everything in the assembly one-liners we are producing.</p>
</div>
<div class="paragraph">
<p>We ignore the <code>\subfunc</code> since it is always 0 on the ops that interest us.</p>
</div>
</div>
<div class="sect4">
<h5 id="m5op-annotations"><a class="anchor" href="#m5op-annotations"></a><a class="link" href="#m5op-annotations">24.8.2.3. m5op annotations</a></h5>
<div class="paragraph">
<p><code>include/gem5/asm/generic/m5ops.h</code> also describes some annotation instructions.</p>
</div>
<div class="paragraph">
<p>What they mean: <a href="https://stackoverflow.com/questions/50583962/what-are-the-gem5-annotations-mops-magic-instructions-and-how-to-use-them" class="bare">https://stackoverflow.com/questions/50583962/what-are-the-gem5-annotations-mops-magic-instructions-and-how-to-use-them</a></p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-arm-linux-kernel-patches"><a class="anchor" href="#gem5-arm-linux-kernel-patches"></a><a class="link" href="#gem5-arm-linux-kernel-patches">24.9. gem5 arm Linux kernel patches</a></h3>
<div class="paragraph">
<p><a href="https://gem5.googlesource.com/arm/linux/" class="bare">https://gem5.googlesource.com/arm/linux/</a> contains an ARM Linux kernel forks with a few gem5 specific Linux kernel patches on top of mainline created by ARM Holdings on top of a few upstream kernel releases.</p>
</div>
<div class="paragraph">
<p>Our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build">build</a> script automatically adds that remote for us as <code>gem5-arm</code>.</p>
</div>
<div class="paragraph">
<p>The patches are optional: the vanilla kernel does boot. But they add some interesting gem5-specific optimizations, instrumentations and device support.</p>
</div>
<div class="paragraph">
<p>The patches also <a href="#notable-alternate-gem5-kernel-configs">add defconfigs</a> that are known to work well with gem5.</p>
</div>
<div class="paragraph">
<p>E.g. for arm v4.9 there is: <a href="https://gem5.googlesource.com/arm/linux/+/917e007a4150d26a0aa95e4f5353ba72753669c7/arch/arm/configs/gem5_defconfig" class="bare">https://gem5.googlesource.com/arm/linux/+/917e007a4150d26a0aa95e4f5353ba72753669c7/arch/arm/configs/gem5_defconfig</a>.</p>
</div>
<div class="paragraph">
<p>In order to use those patches and their associated configs, and, we recommend using <a href="#linux-kernel-build-variants">Linux kernel build variants</a> as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git -C "$(./getvar linux_source_dir)" fetch gem5-arm:gem5/v4.15
git -C "$(./getvar linux_source_dir)" checkout gem5/v4.15
./build-linux \
  --arch aarch64 \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
git -C "$(./getvar linux_source_dir)" checkout -
./run \
  --arch aarch64 \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
;</pre>
</div>
</div>
<div class="paragraph">
<p>QEMU also boots that kernel successfully:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --linux-build-id gem5-v4.15 \
;</pre>
</div>
</div>
<div class="paragraph">
<p>but glibc kernel version checks make init fail with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FATAL: kernel too old</pre>
</div>
</div>
<div class="paragraph">
<p>because glibc was built to expect a newer Linux kernel as shown at: <a href="#fatal-kernel-too-old-failure-in-userland-simulation">Section 11.4.1, &#8220;FATAL: kernel too old failure in userland simulation&#8221;</a>. Your choices to solve this are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>see if there is a more recent gem5 kernel available, or port your patch of interest to the newest kernel</p>
</li>
<li>
<p>modify this repo to use <a href="#libc-choice">uClibc</a>, which is not hard because of Buildroot</p>
</li>
<li>
<p>patch glibc to remove that check, which is easy because glibc is in a submodule of this repo</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It is obviously not possible to understand what the Linux kernel fork commits actually do from their commit message, so let&#8217;s explain them one by one here as we understand them:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>drm: Add component-aware simple encoder</code> allows you to see images through VNC, see: <a href="#gem5-graphic-mode">Section 14.3, &#8220;gem5 graphic mode&#8221;</a></p>
</li>
<li>
<p><code>gem5: Add support for gem5&#8217;s extended GIC mode</code> adds support for more than 8 cores, see: <a href="#gem5-arm-full-system-with-more-than-8-cores">Section 24.3.1.2, &#8220;gem5 ARM full system with more than 8 cores&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.</p>
</div>
<div class="sect3">
<h4 id="gem5-arm-linux-kernel-patches-boot-speedup"><a class="anchor" href="#gem5-arm-linux-kernel-patches-boot-speedup"></a><a class="link" href="#gem5-arm-linux-kernel-patches-boot-speedup">24.9.1. gem5 arm Linux kernel patches boot speedup</a></h4>
<div class="paragraph">
<p>We have observed that with the kernel patches, boot is 2x faster, falling from 1m40s to 50s.</p>
</div>
<div class="paragraph">
<p>With <a href="https://stackoverflow.com/questions/49797246/how-to-monitor-for-how-much-time-each-line-of-stdout-was-the-last-output-line-in/49797547#49797547"><code>ts</code></a>, we see that a large part of the difference is at the message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>clocksource: Switched to clocksource arch_sys_counter</pre>
</div>
</div>
<div class="paragraph">
<p>which takes 4s on the patched kernel, and 30s on the unpatched one! TODO understand why, especially if it is a config difference, or if it actually comes from a patch.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="m5out-directory"><a class="anchor" href="#m5out-directory"></a><a class="link" href="#m5out-directory">24.10. m5out directory</a></h3>
<div class="paragraph">
<p>When you run gem5, it generates an <code>m5out</code> directory at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo $(./getvar --arch arm --emulator gem5 m5out_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>The location of that directory can be set with <code>./gem5.opt -d</code>, and defaults to <code>./m5out</code>.</p>
</div>
<div class="paragraph">
<p>The files in that directory contains some very important information about the run, and you should become familiar with every one of them.</p>
</div>
<div class="sect3">
<h4 id="gem5-m5out-system-terminal-file"><a class="anchor" href="#gem5-m5out-system-terminal-file"></a><a class="link" href="#gem5-m5out-system-terminal-file">24.10.1. gem5 m5out/system.terminal file</a></h4>
<div class="paragraph">
<p>Contains UART output, both from the Linux kernel or from the baremetal system.</p>
</div>
<div class="paragraph">
<p>Can also be seen live on <a href="#m5term">m5term</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-m5out-system-dmesg-file"><a class="anchor" href="#gem5-m5out-system-dmesg-file"></a><a class="link" href="#gem5-m5out-system-dmesg-file">24.10.2. gem5 <code>m5out/system.workload.dmesg</code> file</a></h4>
<div class="paragraph">
<p>This file used to be called just <code>m5out/system.dmesg</code>, but the name was changed after the workload refactorings of March 2020.</p>
</div>
<div class="paragraph">
<p>This file is capable of showing terminal messages that are <code>printk</code> before the serial is enabled as described at: <a href="#linux-kernel-early-boot-messages">Linux kernel early boot messages</a>.</p>
</div>
<div class="paragraph">
<p>The file is dumped only on kernel panics which gem5 can detect by the PC address: <a href="#exit-gem5-on-panic">Exit gem5 on panic</a>.</p>
</div>
<div class="paragraph">
<p>This mechanism can be very useful to debug the Linux kernel boot if problems happen before the serial is enabled.</p>
</div>
<div class="paragraph">
<p>This magic mechanism works by activating an event when the PC reaches the <code>printk</code> address, much like gem5 <a href="#exit-gem5-on-panic">can detect <code>panic</code> by PC</a> and then parsing printk function arguments and buffers!</p>
</div>
<div class="paragraph">
<p>The relevant source is at <a href="https://github.com/gem5/gem5/blob/cd69bb50414450c3bb5ef41dce676b75fd42c0ee/src/kern/linux/printk.cc"><code>src/kern/linux/printk.c</code></a>.</p>
</div>
<div class="paragraph">
<p>We can test this mechanism in a controlled way by hacking a <code>panic()</code> into the kernel next to a <code>printk</code> that shows up before the serial is enabled, e.g. on Linux v5.4.3 we could do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index f296d89be757..3e79916322c2 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6207,6 +6207,7 @@ void __init ftrace_init(void)

    pr_info("ftrace: allocating %ld entries in %ld pages\n",
        count, count / ENTRIES_PER_PAGE + 1);
+   panic("foobar");

    last_ftrace_enabled = ftrace_enabled = 1;</pre>
</div>
</div>
<div class="paragraph">
<p>With this, after the panic, <code>system.workload.dmesg</code> contains on LKMC d09a0d97b81582cc88381c4112db631da61a048d aarch64:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]
[0.000000] Linux version 5.4.3-dirty (lkmc@f7688b48ac46e9a669e279f1bc167722d5141eda) (gcc version 8.3.0 (Buildroot 2019.11-00002-g157ac499cf)) #1 SMP Thu Jan 1 00:00:00 UTC 1970
[0.000000] Machine model: V2P-CA15
[0.000000] Memory limited to 256MB
[0.000000] efi: Getting EFI parameters from FDT:
[0.000000] efi: UEFI not found.
[0.000000] On node 0 totalpages: 65536
[0.000000]   DMA32 zone: 1024 pages used for memmap
[0.000000]   DMA32 zone: 0 pages reserved
[0.000000]   DMA32 zone: 65536 pages, LIFO batch:15
[0.000000] percpu: Embedded 29 pages/cpu s79960 r8192 d30632 u118784
[0.000000] pcpu-alloc: s79960 r8192 d30632 u118784 alloc=29*4096
[0.000000] pcpu-alloc: [0] 0
[0.000000] Detected PIPT I-cache on CPU0
[0.000000] CPU features: detected: ARM erratum 832075
[0.000000] CPU features: detected: EL2 vector hardening
[0.000000] ARM_SMCCC_ARCH_WORKAROUND_1 missing from firmware
[0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64512
[0.000000] Kernel command line: earlyprintk=pl011,0x1c090000 lpj=19988480 rw loglevel=8 mem=256MB root=/dev/sda console_msg_format=syslog nokaslr norandmaps panic=-1 printk.devkmsg=on printk.time=y rw console=ttyAMA0 - lkmc_home=/lkmc
[0.000000] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[0.000000] Memory: 233432K/262144K available (6652K kernel code, 792K rwdata, 2176K rodata, 896K init, 659K bss, 28712K reserved, 0K cma-reserved)
[0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[0.000000] ftrace: allocating 22067 entries in 87 pages</pre>
</div>
</div>
<div class="paragraph">
<p>So we see that messages up to the <code>ftrace</code> do show up!</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-m5out-stats-txt-file"><a class="anchor" href="#gem5-m5out-stats-txt-file"></a><a class="link" href="#gem5-m5out-stats-txt-file">24.10.3. gem5 m5out/stats.txt file</a></h4>
<div class="paragraph">
<p>This file contains important statistics about the run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar --arch aarch64 m5out_dir)/stats.txt"</pre>
</div>
</div>
<div class="paragraph">
<p>Whenever we run <code>m5 dumpstats</code> or when fs.py and se.py are exiting (TODO other scripts?), a section with the following format is added to that file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>---------- Begin Simulation Statistics ----------
[the stats]
---------- End Simulation Statistics   ----------</pre>
</div>
</div>
<div class="paragraph">
<p>That file contains several important execution metrics, e.g. number of cycles and several types of cache misses:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system.cpu.numCycles
system.cpu.dtb.inst_misses
system.cpu.dtb.inst_hits</pre>
</div>
</div>
<div class="paragraph">
<p>For x86, it is interesting to try and correlate <code>numCycles</code> with:</p>
</div>
<div class="paragraph">
<p>In LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df, the stat file for a <a href="#c">C</a> hello world:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>which has a single dump done at the exit, has size 59KB and stat lines of form:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>final_tick                                   91432000                       # Number of ticks from beginning of simulation (restored from checkpoints and never reset)</pre>
</div>
</div>
<div class="paragraph">
<p>We can reduce the file size by adding the <code>?desc=False</code> magic suffix to the stat flie name:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--stats-file stats.txt?desc=false</pre>
</div>
</div>
<div class="paragraph">
<p>as explained in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gem5.opt --stats-help</pre>
</div>
</div>
<div class="paragraph">
<p>and this reduces the file size to 39KB by removing those excessive comments:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>final_tick                                   91432000</pre>
</div>
</div>
<div class="paragraph">
<p>although trailing spaces are still prse</p>
</div>
<div class="paragraph">
<p>We can further reduce this size by removing spaces from the dumps with this hack:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>         ccprintf(stream, " |%12s %10s %10s",
                  ValueToString(value, precision), pdfstr.str(), cdfstr.str());
     } else {
-        ccprintf(stream, "%-40s %12s %10s %10s", name,
-                 ValueToString(value, precision), pdfstr.str(), cdfstr.str());
+        ccprintf(stream, "%s %s", name, ValueToString(value, precision));
+        if (pdfstr.rdbuf()-&gt;in_avail())
+            stream &lt;&lt; " " &lt;&lt; pdfstr.str();
+        if (cdfstr.rdbuf()-&gt;in_avail())
+            stream &lt;&lt; " " &lt;&lt; cdfstr.str();

         if (descriptions) {
             if (!desc.empty())</pre>
</div>
</div>
<div class="paragraph">
<p>and after that the file size went down to 21KB.</p>
</div>
<div class="sect4">
<h5 id="gem5-hdf5-statistics"><a class="anchor" href="#gem5-hdf5-statistics"></a><a class="link" href="#gem5-hdf5-statistics">24.10.3.1. gem5 HDF5 statistics</a></h5>
<div class="paragraph">
<p>We can make gem5 dump statistics in the <a href="#hdf5">HDF5</a> format by adding the magic <code>h5://</code> prefix to the file name as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gem5.opt --stats-file h5://stats.h5</pre>
</div>
</div>
<div class="paragraph">
<p>as explained in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gem5.opt --stats-help</pre>
</div>
</div>
<div class="paragraph">
<p>This is not exposed in LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 however, you just have to <a href="#dry-run">hack the gem5 CLI for now</a>.</p>
</div>
<div class="paragraph">
<p>TODO what is the advantage? The generated file for <code>--stats-file h5://stats.h5?desc=False</code> in LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df for a single dump was 946K, so much larger than the text version seen at <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a> which was only 59KB max!</p>
</div>
<div class="paragraph">
<p>We then try to see if it is any better when you have a bunch of dump events:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/m5ops.c --cli-args 'd 1000'</pre>
</div>
</div>
<div class="paragraph">
<p>and there yes, we see that the file size fell from 39MB on <code>stats.txt</code> to 3.2MB on <code>stats.m5</code>, so the increase observed previously was just due to some initial size overhead (considering the patched gem5 with no spaces in the text file).</p>
</div>
<div class="paragraph">
<p>We also note however that the stat dump made the such a simulation that just loops and dumps considerably slower, from 3s to 15s on <a href="#p51">2017 Lenovo ThinkPad P51</a>. Fascinating, we are definitely not disk bound there.</p>
</div>
<div class="paragraph">
<p>We enable HDF5 on the build by default with <code>USE_HDF5=1</code>. To disable it, you can add <code>USE_HDF5=0</code> to the build as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 -- USE_HDF5=0</pre>
</div>
</div>
<div class="paragraph">
<p>Library support is automatically detected, and only built if you have it installed. But there have been some compilation bugs with HDF5, which is why you might want to turn it off sometimes, e.g.: <a href="https://gem5.atlassian.net/browse/GEM5-365" class="bare">https://gem5.atlassian.net/browse/GEM5-365</a></p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-only-dump-selected-stats"><a class="anchor" href="#gem5-only-dump-selected-stats"></a><a class="link" href="#gem5-only-dump-selected-stats">24.10.3.2. gem5 only dump selected stats</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certain-selected-stats-in-gem5" class="bare">https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certain-selected-stats-in-gem5</a></p>
</div>
<div class="paragraph">
<p>To prevent the stats file from becoming humongous.</p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certain-selected-stats-in-gem5/57221132#57221132" class="bare">https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certain-selected-stats-in-gem5/57221132#57221132</a></p>
</div>
</div>
<div class="sect4">
<h5 id="meaning-of-each-gem5-stat"><a class="anchor" href="#meaning-of-each-gem5-stat"></a><a class="link" href="#meaning-of-each-gem5-stat">24.10.3.3. Meaning of each gem5 stat</a></h5>
<div class="paragraph">
<p>Well, run minimal examples, and reverse engineer them up!</p>
</div>
<div class="paragraph">
<p>We can start with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/freestanding/linux/hello.S">userland/arch/x86_64/freestanding/linux/hello.S</a> on atomic with <a href="#gem5-execall-trace-format">gem5 ExecAll trace format</a>.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace ExecAll \
  --trace-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: system.cpu: A0 T0 : @_start    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
    500: system.cpu: A0 T0 : @_start+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
   1000: system.cpu: A0 T0 : @_start+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)
   1500: system.cpu: A0 T0 : @_start+12    :   movz   x8, #64, #0       : IntAlu :  D=0x0000000000000040  flags=(IsInteger)
   2000: system.cpu: A0 T0 : @_start+16    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
   2500: system.cpu: A0 T0 : @_start+20    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   3000: system.cpu: A0 T0 : @_start+24    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
   3500: system.cpu: A0 T0 : @_start+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)</pre>
</div>
</div>
<div class="paragraph">
<p>The most important stat of all is usually the cycle count, which is a direct measure of performance if you modelled you system well:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sim_ticks 3500 # Number of ticks simulated</pre>
</div>
</div>
<div class="paragraph">
<p>Next, <code>sim_insts</code> and <code>sim_ops</code> are often critical:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sim_insts 6 # Number of instructions simulated
sim_ops   6 # Number of ops (including micro ops) simulated</pre>
</div>
</div>
<div class="paragraph">
<p><code>sim_ops</code> is like <code>sim_insts</code> but it also includes <a href="#gem5-microops">gem5 microops</a>.</p>
</div>
<div class="paragraph">
<p>In <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a>, syscall instructions are magic, and therefore appear to not be counted, that is why we get 6 instructions instead of 8.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-stats-internals"><a class="anchor" href="#gem5-stats-internals"></a><a class="link" href="#gem5-stats-internals">24.10.3.4. gem5 stats internals</a></h5>
<div class="paragraph">
<p>This describes the internals of the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a>.</p>
</div>
<div class="paragraph">
<p>GDB call stack to <code>dumpstats</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Stats::pythonDump () at build/ARM/python/pybind11/stats.cc:58
Stats::StatEvent::process() ()
GlobalEvent::BarrierEvent::process (this=0x555559fa6a80) at build/ARM/sim/global_event.cc:131
EventQueue::serviceOne (this=this@entry=0x555558c36080) at build/ARM/sim/eventq.cc:228
doSimLoop (eventq=0x555558c36080) at build/ARM/sim/simulate.cc:219
simulate (num_cycles=&lt;optimized out&gt;) at build/ARM/sim/simulate.cc:132</pre>
</div>
</div>
<div class="paragraph">
<p><code>Stats::pythonDump</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
pythonDump()
{
    py::module m = py::module::import("m5.stats");
    m.attr("dump")();
}</pre>
</div>
</div>
<div class="paragraph">
<p>This calls <code>src/python/m5/stats/<em>init</em>.py</code> in <code>def dump</code> does the main dumping</p>
</div>
<div class="paragraph">
<p>That function does notably:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    for output in outputList:
        if output.valid():
            output.begin()
            for stat in stats_list:
                stat.visit(output)
            output.end()</pre>
</div>
</div>
<div class="paragraph">
<p><code>begin</code> and <code>end</code> are defined in C++ and output the header and tail respectively</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
Text::begin()
{
    ccprintf(*stream, "\n---------- Begin Simulation Statistics ----------\n");
}

void
Text::end()
{
    ccprintf(*stream, "\n---------- End Simulation Statistics   ----------\n");
    stream-&gt;flush();
}</pre>
</div>
</div>
<div class="paragraph">
<p><code>stats_list</code> contains the stats, and <code>stat.visit</code> prints them, <code>outputList</code> contains by default just the text output. I don&#8217;t see any other types of output in gem5, but likely JSON / binary formats could be envisioned.</p>
</div>
<div class="paragraph">
<p>Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-config-ini"><a class="anchor" href="#gem5-config-ini"></a><a class="link" href="#gem5-config-ini">24.10.4. gem5 config.ini</a></h4>
<div class="paragraph">
<p>The <code>m5out/config.ini</code> file, contains a very good high level description of the system:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less $(./getvar --arch arm --emulator gem5 m5out_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>That file contains a tree representation of the system, sample excerpt:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[root]
type=Root
children=system
full_system=true

[system]
type=ArmSystem
children=cpu cpu_clk_domain
auto_reset_addr_64=false
semihosting=Null

[system.cpu]
type=AtomicSimpleCPU
children=dstage2_mmu dtb interrupts isa istage2_mmu itb tracer
branchPred=Null

[system.cpu_clk_domain]
type=SrcClockDomain
clock=500</pre>
</div>
</div>
<div class="paragraph">
<p>Each node has:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a list of child nodes, e.g. <code>system</code> is a child of <code>root</code>, and both <code>cpu</code> and <code>cpu_clk_domain</code> are children of <code>system</code></p>
</li>
<li>
<p>a list of parameters, e.g. <code>system.semihosting</code> is <code>Null</code>, which means that <a href="#semihosting">Semihosting</a> was turned off</p>
<div class="ulist">
<ul>
<li>
<p>the <code>type</code> parameter shows is present on every node, and it maps to a <code>Python</code> object that inherits from <a href="#gem5-python-c-interaction"><code>SimObject</code></a>.</p>
<div class="paragraph">
<p>For example, <code>AtomicSimpleCPU</code> maps is defined at <a href="https://github.com/gem5/gem5/blob/05c4c2b566ce351ab217b2bd7035562aa7a76570/src/cpu/simple/AtomicSimpleCPU.py#L45">src/cpu/simple/AtomicSimpleCPU.py</a>.</p>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Set custom configs with the <code>--param</code> option of <code>fs.py</code>, e.g. we can make gem5 wait for GDB to connect with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fs.py --param 'system.cpu[0].wait_for_remote_gdb = True'</pre>
</div>
</div>
<div class="paragraph">
<p>More complex settings involving new classes however require patching the config files, although it is easy to hack this up. See for example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/patches/manual/gem5-semihost.patch">patches/manual/gem5-semihost.patch</a>.</p>
</div>
<div class="paragraph">
<p>Modifying the <code>config.ini</code> file manually does nothing since it gets overwritten every time.</p>
</div>
<div class="sect4">
<h5 id="gem5-config-dot"><a class="anchor" href="#gem5-config-dot"></a><a class="link" href="#gem5-config-dot">24.10.4.1. gem5 config.dot</a></h5>
<div class="paragraph">
<p>The <code>m5out/config.dot</code> file contains a graphviz <code>.dot</code> file that provides a simplified graphical view of a subset of the <a href="#gem5-config-ini">gem5 config.ini</a>.</p>
</div>
<div class="paragraph">
<p>This file gets automatically converted to <code>.svg</code> and <code>.pdf</code>, which you can view after running gem5 with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.pdf"
xdg-open "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot.svg"</pre>
</div>
</div>
<div class="paragraph">
<p>An example of such file can be seen at: <a href="#config-dot-svg-timingsimplecpu"><code>config.dot.svg</code> for a TimingSimpleCPU without caches.</a>.</p>
</div>
<div class="paragraph">
<p>On Ubuntu 20.04, you can also see the dot file "directly" with xdot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>xdot "$(./getvar --arch arm --emulator gem5 m5out_dir)/config.dot"</pre>
</div>
</div>
<div class="paragraph">
<p>which is kind of really cool because it allows you to view graph arrows on hover. This can be very useful because the PDF and SVG often overlap so many arrows together that you just can&#8217;t know which one is coming from/going to where.</p>
</div>
<div class="paragraph">
<p>It is worth noting that if you are running a bunch of short simulations, dot/SVG/PDF generation could have a significant impact in simulation startup time, so it is something to watch out for. As per <a href="https://gem5-review.googlesource.com/c/public/gem5/+/29232" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/29232</a> it can be turned off with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gem5.opt --dot-config=''</pre>
</div>
</div>
<div class="paragraph">
<p>or in LKMC:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gem5-exe-args='--dot-config= --json-config= --dump-config='</pre>
</div>
</div>
<div class="paragraph">
<p>The time difference can be readily observed on minimal examples by running gem5 with <code>time</code>.</p>
</div>
<div class="paragraph">
<p>By looking into gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 <code>src/python/m5/util/dot_writer.py</code> are can try to remove the SVG/PDF conversion to see if those dominate the runtime:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def do_dot(root, outdir, dotFilename):
    if not pydot:
        warn("No dot file generated. " +
             "Please install pydot to generate the dot file and pdf.")
        return
    # * use ranksep &gt; 1.0 for for vertical separation between nodes
    # especially useful if you need to annotate edges using e.g. visio
    # which accepts svg format
    # * no need for hoizontal separation as nothing moves horizonally
    callgraph = pydot.Dot(graph_type='digraph', ranksep='1.3')
    dot_create_nodes(root, callgraph)
    dot_create_edges(root, callgraph)
    dot_filename = os.path.join(outdir, dotFilename)
    callgraph.write(dot_filename)
    try:
        # dot crashes if the figure is extremely wide.
        # So avoid terminating simulation unnecessarily
        callgraph.write_svg(dot_filename + ".svg")
        callgraph.write_pdf(dot_filename + ".pdf")
    except:
        warn("failed to generate dot output from %s", dot_filename)</pre>
</div>
</div>
<div class="paragraph">
<p>but nope, they don&#8217;t, <code>dot_create_nodes</code> and <code>dot_create_edges</code> are the culprits, so the only way to gain speed is to remove <code>.dot</code> generation altogether. It is tempting to do this by default on LKMC and add an option to enable dot generation when desired so we can be a bit faster by default&#8230;&#8203; but I&#8217;m lazy to document the option right now. When it annoys me further maybe :-)</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="m5term"><a class="anchor" href="#m5term"></a><a class="link" href="#m5term">24.11. m5term</a></h3>
<div class="paragraph">
<p>We use the <code>m5term</code> in-tree executable to connect to the terminal instead of a direct <code>telnet</code>.</p>
</div>
<div class="paragraph">
<p>If you use <code>telnet</code> directly, it mostly works, but certain interactive features don&#8217;t, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>up and down arrows for history navigation</p>
</li>
<li>
<p>tab to complete paths</p>
</li>
<li>
<p><code>Ctrl-C</code> to kill processes</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO understand in detail what <code>m5term</code> does differently than <code>telnet</code>.</p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-python-scripts-without-rebuild"><a class="anchor" href="#gem5-python-scripts-without-rebuild"></a><a class="link" href="#gem5-python-scripts-without-rebuild">24.12. gem5 Python scripts without rebuild</a></h3>
<div class="paragraph">
<p>We have made a crazy setup that allows you to just <code>cd</code> into <code>submodules/gem5</code>, and edit Python scripts directly there.</p>
</div>
<div class="paragraph">
<p>This is not normally possible with Buildroot, since normal Buildroot packages first copy files to the output directory (<code>$(./getvar -a &lt;arch&gt; buildroot_build_build_dir)/&lt;pkg&gt;</code>), and then build there.</p>
</div>
<div class="paragraph">
<p>So if you modified the Python scripts with this setup, you would still need to <code>./build</code> to copy the modified files over.</p>
</div>
<div class="paragraph">
<p>For gem5 specifically however, we have hacked up the build so that we <code>cd</code> into the <code>submodules/gem5</code> tree, and then do an <a href="https://stackoverflow.com/questions/54343515/how-to-build-gem5-out-of-tree/54343516#54343516">out of tree</a> build to <code>out/common/gem5</code>.</p>
</div>
<div class="paragraph">
<p>Another advantage of this method is the we factor out the <code>arm</code> and <code>aarch64</code> gem5 builds which are identical and large, as well as the smaller arch generic pieces.</p>
</div>
<div class="paragraph">
<p>Using Buildroot for gem5 is still convenient because we use it to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>to cross build <code>m5</code> for us</p>
</li>
<li>
<p>check timestamps and skip the gem5 build when it is not requested</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The out of build tree is required, because otherwise Buildroot would copy the output build of all archs to each arch directory, resulting in <code>arch^2</code> build copies, which is significant.</p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-fs-biglittle"><a class="anchor" href="#gem5-fs-biglittle"></a><a class="link" href="#gem5-fs-biglittle">24.13. gem5 fs_bigLITTLE</a></h3>
<div class="paragraph">
<p>By default, we use <code>configs/example/fs.py</code> script.</p>
</div>
<div class="paragraph">
<p>The <code>--gem5-script biglittle</code> option enables the alternative <code>configs/example/arm/fs_bigLITTLE.py</code> script instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --gem5-script biglittle</pre>
</div>
</div>
<div class="paragraph">
<p>Advantages over <code>fs.py</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>more representative of mobile ARM SoCs, which almost always have big little cluster</p>
</li>
<li>
<p>simpler than <code>fs.py</code>, and therefore easier to understand and modify</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Disadvantages over <code>fs.py</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>only works for ARM, not other archs</p>
</li>
<li>
<p>not as many configuration options as <code>fs.py</code>, many things are hardcoded</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We setup 2 big and 2 small CPUs, but <code>cat /proc/cpuinfo</code> shows 4 identical CPUs instead of 2 of two different types, likely because gem5 does not expose some informational register much like the caches: <a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15426.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15426.html</a> <a href="#gem5-config-ini">gem5 config.ini</a> does show that the two big ones are <code>DerivO3CPU</code> and the small ones are <code>MinorCPU</code>.</p>
</div>
<div class="paragraph">
<p>TODO: why is the <code>--dtb</code> required despite <code>fs_bigLITTLE.py</code> having a DTB generation capability? Without it, nothing shows on terminal, and the simulation terminates with <code>simulate() limit reached  @  18446744073709551615</code>. The magic <code>vmlinux.vexpress_gem5_v1.20170616</code> works however without a DTB.</p>
</div>
<div class="paragraph">
<p>Tested on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/18c1c823feda65f8b54cd38e261c282eee01ed9f">18c1c823feda65f8b54cd38e261c282eee01ed9f</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-in-tree-tests"><a class="anchor" href="#gem5-in-tree-tests"></a><a class="link" href="#gem5-in-tree-tests">24.14. gem5 in-tree tests</a></h3>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52279971/how-to-run-the-gem5-unit-tests" class="bare">https://stackoverflow.com/questions/52279971/how-to-run-the-gem5-unit-tests</a></p>
</div>
<div class="paragraph">
<p>All those tests could in theory be added to this repo instead of to gem5, and this is actually the superior setup as it is cross emulator.</p>
</div>
<div class="paragraph">
<p>But can the people from the project be convinced of that?</p>
</div>
<div class="sect3">
<h4 id="gem5-unit-tests"><a class="anchor" href="#gem5-unit-tests"></a><a class="link" href="#gem5-unit-tests">24.14.1. gem5 unit tests</a></h4>
<div class="paragraph">
<p>These are just very small GTest tests that test a single class in isolation, they don&#8217;t run any executables.</p>
</div>
<div class="paragraph">
<p>Build the unit tests and run them:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --unit-tests</pre>
</div>
</div>
<div class="paragraph">
<p>Running individual unit tests is not yet exposed, but it is easy to do: while running the full tests, GTest prints each test command being run, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/path/to/build/ARM/base/circlebuf.test.opt --gtest_output=xml:/path/to/build/ARM/unittests.opt/base/circlebuf.test.xml
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from CircleBufTest
[ RUN      ] CircleBufTest.BasicReadWriteNoOverflow
[       OK ] CircleBufTest.BasicReadWriteNoOverflow (0 ms)
[ RUN      ] CircleBufTest.SingleWriteOverflow
[       OK ] CircleBufTest.SingleWriteOverflow (0 ms)
[ RUN      ] CircleBufTest.MultiWriteOverflow
[       OK ] CircleBufTest.MultiWriteOverflow (0 ms)
[ RUN      ] CircleBufTest.PointerWrapAround
[       OK ] CircleBufTest.PointerWrapAround (0 ms)
[----------] 4 tests from CircleBufTest (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 4 tests.</pre>
</div>
</div>
<div class="paragraph">
<p>so you can just copy paste the command.</p>
</div>
<div class="paragraph">
<p>Building individual tests is possible with <code>--unit-test</code> (singular, no 's'):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --unit-test base/circlebuf.test</pre>
</div>
</div>
<div class="paragraph">
<p>This does not run the test however.</p>
</div>
<div class="paragraph">
<p>Note that the command and it&#8217;s corresponding results don&#8217;t need to show consecutively on stdout because tests are run in parallel. You just have to match them based on the class name <code>CircleBufTest</code> to the file <code>circlebuf.test.cpp</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-regression-tests"><a class="anchor" href="#gem5-regression-tests"></a><a class="link" href="#gem5-regression-tests">24.14.2. gem5 regression tests</a></h4>
<div class="paragraph">
<p>This section is about running the gem5 in-tree tests.</p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52279971/how-to-run-the-gem5-unit-tests" class="bare">https://stackoverflow.com/questions/52279971/how-to-run-the-gem5-unit-tests</a></p>
</div>
<div class="paragraph">
<p>Running the larger 2019 regression tests is exposed for example with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --arch aarch64
./gem5-regression --arch aarch64 -- --length quick --length long</pre>
</div>
</div>
<div class="paragraph">
<p>Sample run time: 87 minutes on <a href="#p51">2017 Lenovo ThinkPad P51</a> Ubuntu 20.04 gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1.</p>
</div>
<div class="paragraph">
<p>After the first run has downloaded the test binaries for you, you can speed up the process a little bit by skipping an useless SCons call:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-regression --arch aarch64 -- --length quick --length long --skip-build</pre>
</div>
</div>
<div class="paragraph">
<p>Note however that running without <code>--skip-build</code> is required at least once to download the test binaries, because the test interface is bad.</p>
</div>
<div class="paragraph">
<p>List available instead of running them:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-regression --arch aarch64 --cmd list -- --length quick --length long</pre>
</div>
</div>
<div class="paragraph">
<p>You can then pick one suite (has to be a suite, not an "individual test") from the list and run just it e.g. with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./gem5-regression --arch aarch64 -- --uid SuiteUID:tests/gem5/cpu_tests/test.py:cpu_test_AtomicSimpleCPU_Bubblesort-ARM-opt</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-simulate-limit-reached"><a class="anchor" href="#gem5-simulate-limit-reached"></a><a class="link" href="#gem5-simulate-limit-reached">24.15. gem5 simulate() limit reached</a></h3>
<div class="paragraph">
<p>This error happens when the following instruction limits are reached:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system.cpu[0].max_insts_all_threads
system.cpu[0].max_insts_any_thread</pre>
</div>
</div>
<div class="paragraph">
<p>If the parameter is not set, it defaults to <code>0</code>, which is magic and means the huge maximum value of <code>uint64_t</code>: 0xFFFFFFFFFFFFFFFF, which in practice would require a very long simulation if at least one CPU were live.</p>
</div>
<div class="paragraph">
<p>So this usually means all CPUs are in a sleep state, and no events are scheduled in the future, which usually indicates a bug in either gem5 or guest code, leading gem5 to blow up.</p>
</div>
<div class="paragraph">
<p>Still, fs.py at gem5 08c79a194d1a3430801c04f37d13216cc9ec1da3 does not exit with non-zero status due to this&#8230;&#8203; and so we just parse it out just as for <a href="#m5-fail">m5 fail</a>&#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>A trivial and very direct way to see message would be:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --emulator gem5 \
  --userland userland/arch/x86_64/freestanding/linux/hello.S \
  --trace-insts-stdout \
  -- \
  --param 'system.cpu[0].max_insts_all_threads = 3' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which as of lkmc 402059ed22432bb351d42eb10900e5a8e06aa623 runs only the first three instructions and quits!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: Entering event queue @ 0.  Starting simulation...
      0: system.cpu A0 T0 : @asm_main_after_prologue    : mov   rdi, 0x1
      0: system.cpu A0 T0 : @asm_main_after_prologue.0  :   MOV_R_I : limm   rax, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7    : mov rdi, 0x1
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7.0  :   MOV_R_I : limm   rdi, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14    : lea        rsi, DS:[rip + 0x19]
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14.0  :   LEA_R_P : rdip   t7, %ctrl153,  : IntAlu :  D=0x000000000040008d  flags=(IsInteger|IsMicroop|IsDelayedCommit|IsFirstMicroop)
   2500: system.cpu A0 T0 : @asm_main_after_prologue+14.1  :   LEA_R_P : lea   rsi, DS:[t7 + 0x19] : IntAlu :  D=0x00000000004000a6  flags=(IsInteger|IsMicroop|IsLastMicroop)
Exiting @ tick 3000 because all threads reached the max instruction count</pre>
</div>
</div>
<div class="paragraph">
<p>The exact same can be achieved with the older hardcoded <code>--maxinsts</code> mechanism present in <code>se.py</code> and <code>fs.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --emulator gem5 \
  --userland \userland/arch/x86_64/freestanding/linux/hello.S \
  --trace-insts-stdout \
  -- \
  --maxinsts 3
;</pre>
</div>
</div>
<div class="paragraph">
<p>Other related fs.py options are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>--abs-max-tick</code>: set the maximum guest simulation time. The same scale as the ExecAll trace is used. E.g., for the above example with 3 instructions, the same trace would be achieved with a value of 3000.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The message also shows on <a href="#user-mode-simulation">User mode simulation</a> deadlocks, for example in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_deadlock.c">userland/posix/pthread_deadlock.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --emulator gem5 \
  --userland userland/posix/pthread_deadlock.c \
  --cli-args 1 \
;</pre>
</div>
</div>
<div class="paragraph">
<p>ends in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Exiting @ tick 18446744073709551615 because simulate() limit reached</pre>
</div>
</div>
<div class="paragraph">
<p>where 18446744073709551615 is 0xFFFFFFFFFFFFFFFF in decimal.</p>
</div>
<div class="paragraph">
<p>And there is a <a href="#baremetal">Baremetal</a> example at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/no_bootloader/wfe_loop.S">baremetal/arch/aarch64/no_bootloader/wfe_loop.S</a> that dies on <a href="#arm-wfe-and-sev-instructions">WFE</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --baremetal baremetal/arch/aarch64/no_bootloader/wfe_loop.S \
  --emulator gem5 \
  --trace-insts-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info: Entering event queue @ 0.  Starting simulation...
      0: system.cpu A0 T0 : @lkmc_start    :   wfe                      : IntAlu :  D=0x0000000000000000  flags=(IsSerializeAfter|IsNonSpeculative|IsQuiesce|IsUnverifiable)
   1000: system.cpu A0 T0 : @lkmc_start+4    :   b   &lt;lkmc_start&gt;         : IntAlu :   flags=(IsControl|IsDirectControl|IsUncondControl)
   1500: system.cpu A0 T0 : @lkmc_start    :   wfe                      : IntAlu :  D=0x0000000000000000  flags=(IsSerializeAfter|IsNonSpeculative|IsQuiesce|IsUnverifiable)
Exiting @ tick 18446744073709551615 because simulate() limit reached</pre>
</div>
</div>
<div class="paragraph">
<p>Other examples of the message:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-baremetal-multicore">ARM baremetal multicore</a> with a single CPU stays stopped at an WFE sleep instruction</p>
</li>
<li>
<p>this sample bug on se.py multithreading: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/81" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/81</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="gem5-build-options"><a class="anchor" href="#gem5-build-options"></a><a class="link" href="#gem5-build-options">24.16. gem5 build options</a></h3>
<div class="paragraph">
<p>In order to use different build options, you might also want to use <a href="#gem5-build-variants">gem5 build variants</a> to keep the build outputs separate from one another.</p>
</div>
<div class="sect3">
<h4 id="gem5-debug-build"><a class="anchor" href="#gem5-debug-build"></a><a class="link" href="#gem5-debug-build">24.16.1. gem5 debug build</a></h4>
<div class="paragraph">
<p>How to use it in LKMC: <a href="#debug-the-emulator">Section 23.8, &#8220;Debug the emulator&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>If you build gem5 with <code>scons build/ARM/gem5.debug</code>, then that is a <code>.debug</code> build.</p>
</div>
<div class="paragraph">
<p>It relates to the more common <code>.opt</code> build just as explained at <a href="#debug-the-emulator">Section 23.8, &#8220;Debug the emulator&#8221;</a>: both <code>.opt</code> and <code>.debug</code> have <code>-g</code>, but <code>.opt</code> uses <code>-O2</code> while <code>.debug</code> uses <code>-O0</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-fast-build"><a class="anchor" href="#gem5-fast-build"></a><a class="link" href="#gem5-fast-build">24.16.2. gem5 fast build</a></h4>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --gem5-build-type fast</pre>
</div>
</div>
<div class="paragraph">
<p>How it goes faster is explained at: <a href="https://stackoverflow.com/questions/59860091/how-to-increase-the-simulation-speed-of-a-gem5-run/59861375#59861375" class="bare">https://stackoverflow.com/questions/59860091/how-to-increase-the-simulation-speed-of-a-gem5-run/59861375#59861375</a></p>
</div>
<div class="paragraph">
<p>Disables debug symbols (no <code>-g</code>) for some reason.</p>
</div>
<div class="paragraph">
<p>Benchmarks present at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#benchmark-emulators-on-userland-executables">Section 35.2.2, &#8220;Benchmark emulators on userland executables&#8221;</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gem5-prof-and-perf-builds"><a class="anchor" href="#gem5-prof-and-perf-builds"></a><a class="link" href="#gem5-prof-and-perf-builds">24.16.3. gem5 prof and perf builds</a></h4>
<div class="paragraph">
<p>Profiling builds as of 3cea7d9ce49bda49c50e756339ff1287fd55df77 both use: <code>-g -O3</code> and disable asserts and logging like the <a href="#gem5-fast-build">gem5 fast build</a> and:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>prof</code> uses <code>-pg</code> for gprof</p>
</li>
<li>
<p><code>perf</code> uses <code>-lprofile</code> for google-pprof</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Profiling techniques are discussed in more detail at: <a href="#profiling-userland-programs">Profiling userland programs</a>.</p>
</div>
<div class="paragraph">
<p>For the <code>prof</code> build, you can get the <code>gmon.out</code> file with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --userland userland/c/hello.c --gem5-build-type prof
gprof "$(./getvar --arch aarch64 gem5_executable)" &gt; tmp.gprof</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-clang-build"><a class="anchor" href="#gem5-clang-build"></a><a class="link" href="#gem5-clang-build">24.16.4. gem5 clang build</a></h4>
<div class="paragraph">
<p>TODO test properly, benchmark vs GCC.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install clang
./build-gem5 --gem5-clang
./run --emulator gem5 --gem5-clang</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-sanitation-build"><a class="anchor" href="#gem5-sanitation-build"></a><a class="link" href="#gem5-sanitation-build">24.16.5. gem5 sanitation build</a></h4>
<div class="paragraph">
<p>If there gem5 appears to have a C++ undefined behaviour bug, which is often very difficult to track down, you can try to build it with the following extra SCons options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --gem5-build-id san --verbose -- --with-ubsan --without-tcmalloc</pre>
</div>
</div>
<div class="paragraph">
<p>This will make GCC do a lot of extra sanitation checks at compile and run time.</p>
</div>
<div class="paragraph">
<p>As a result, the build and runtime will be way slower than normal, but that still might be the fastest way to solve undefined behaviour problems.</p>
</div>
<div class="paragraph">
<p>Ideally, we should also be able to run it with asan with <code>--with-asan</code>, but if we try then the build fails at gem5 16eeee5356585441a49d05c78abc328ef09f7ace (with two ubsan trivial fixes I&#8217;ll push soon):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>=================================================================
==9621==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 371712 byte(s) in 107 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff03950d065 in dictresize ../Objects/dictobject.c:643

Direct leak of 23728 byte(s) in 26 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff03945e40d in _PyObject_GC_Malloc ../Modules/gcmodule.c:1499
    #2 0x7ff03945e40d in _PyObject_GC_Malloc ../Modules/gcmodule.c:1493

Direct leak of 2928 byte(s) in 43 object(s) allocated from:
    #0 0x7ff03980487e in __interceptor_realloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c87e)
    #1 0x7ff03951d763 in list_resize ../Objects/listobject.c:62
    #2 0x7ff03951d763 in app1 ../Objects/listobject.c:277
    #3 0x7ff03951d763 in PyList_Append ../Objects/listobject.c:289

Direct leak of 2002 byte(s) in 3 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff0394fd813 in PyString_FromStringAndSize ../Objects/stringobject.c:88
    #2 0x7ff0394fd813 in PyString_FromStringAndSize ../Objects/stringobject.c:
    Direct leak of 40 byte(s) in 2 object(s) allocated from
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff03951ea4b in PyList_New ../Objects/listobject.c:152

Indirect leak of 10384 byte(s) in 11 object(s) allocated from
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448
    #1 0x7ff03945e40d in _PyObject_GC_Malloc ../Modules/gcmodule.c:
    #2 0x7ff03945e40d in _PyObject_GC_Malloc ../Modules/gcmodule.c:1493

Indirect leak of 4089 byte(s) in 6 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff0394fd648 in PyString_FromString ../Objects/stringobject.c:143

Indirect leak of 2090 byte(s) in 3 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448
    #1 0x7ff0394eb36f in type_new ../Objects/typeobject.c:
    #2 0x7ff0394eb36f in type_new ../Objects/typeobject.c:2094
Indirect leak of 1346 byte(s) in 2 object(s) allocated from:
    #0 0x7ff039804448 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x7ff0394fd813 in PyString_FromStringAndSize ../Objects/stringobject.c:
    #2 0x7ff0394fd813 in PyString_FromStringAndSize ../Objects/stringobject.c:
    SUMMARY: AddressSanitizer: 418319 byte(s) leaked in 203 allocation(s).</pre>
</div>
</div>
<div class="paragraph">
<p>From the message, this appears however to be a Python / pyenv11 bug however and not in gem5 specifically. I think it worked when I tried it in the past in an older gem5 / Ubuntu.</p>
</div>
<div class="paragraph">
<p><code>--without-tcmalloc</code> is needed / a good idea when using <code>--with-asan</code>: <a href="https://stackoverflow.com/questions/42712555/address-sanitizer-fsanitize-address-works-with-tcmalloc" class="bare">https://stackoverflow.com/questions/42712555/address-sanitizer-fsanitize-address-works-with-tcmalloc</a> since both do more or less similar jobs, see also <a href="#memory-leaks">Memory leaks</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-ruby-build"><a class="anchor" href="#gem5-ruby-build"></a><a class="link" href="#gem5-ruby-build">24.16.6. gem5 Ruby build</a></h4>
<div class="paragraph">
<p>gem5 has two types of memory system:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the classic memory system, which is used by default, its caches are covered at: <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a></p>
</li>
<li>
<p>the Ruby memory system</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The Ruby memory system includes the SLICC domain specific language to describe memory systems: <a href="http://gem5.org/Ruby" class="bare">http://gem5.org/Ruby</a> SLICC transpiles to C++ auto-generated files under <code>build/&lt;isa&gt;/mem/ruby/protocol/</code>.</p>
</div>
<div class="paragraph">
<p>Ruby seems to have usage outside of gem5, but the naming overload with the <a href="https://en.wikipedia.org/wiki/Ruby_(programming_language)">Ruby programming language</a>, which also has <a href="https://thoughtbot.com/blog/writing-a-domain-specific-language-in-ruby">domain specific languages</a> as a concept, makes it impossible to google anything about it!</p>
</div>
<div class="paragraph">
<p>Since it is not the default, Ruby is generally less stable that the classic memory model. However, because it allows describing a wide variety of important <a href="#cache-coherence">cache coherence protocols</a>, while the classic system only describes a single protocol, Ruby is very importanonly describes a single protocol, Ruby is a very important feature of gem5.</p>
</div>
<div class="paragraph">
<p>Ruby support must be enabled at compile time with the <code>scons PROTOCOL=</code> flag, which compiles support for the desired memory system type.</p>
</div>
<div class="paragraph">
<p>Note however that most ISAs already implicitly set <code>PROTOCOL</code> via the <code>build_opts/</code> directory, e.g. <code>build_opts/ARM</code> contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>PROTOCOL = 'MOESI_CMP_directory'</pre>
</div>
</div>
<div class="paragraph">
<p>and therefore ARM already compiles <code>MOESI_CMP_directory</code> by default.</p>
</div>
<div class="paragraph">
<p>Then, with <code>fs.py</code> and <code>se.py</code>, you can choose to use either the classic or the ruby system type selected at build time with <code>PROTOCOL=</code> at runtime by passing the <code>--ruby</code> option:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>if <code>--ruby</code> is given, use the ruby memory system that was compiled into gem5. Caches are always present when Ruby is used, since the main goal of Ruby is to specify the cache coherence protocol, and it therefore hardcodes cache hierarchies.</p>
</li>
<li>
<p>otherwise, use the classic memory system. Caches may be optional for certain CPU types and are enabled with <code>--caches</code>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Note that the <code>--ruby</code> option has some crazy side effects besides enabling Ruby, e.g. it <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/configs/ruby/Ruby.py#L61">sets the default <code>--cpu-type</code> to <code>TimingSimpleCPU</code> instead of the otherwise default <code>AtomicSimpleCPU</code></a>. TODO: I have been told that this is because <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">sends the packet atomically,atomic requests do not work with Ruby, only timing</a>.</p>
</div>
<div class="paragraph">
<p>It is not possible to build more than one Ruby system into a single build, and this is a major pain point for testing Ruby: <a href="https://gem5.atlassian.net/browse/GEM5-467" class="bare">https://gem5.atlassian.net/browse/GEM5-467</a></p>
</div>
<div class="paragraph">
<p>For example, to use a two level <a href="#mesi-cache-coherence-protocol">MESI cache coherence protocol</a> we can do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --arch aarch64 --gem5-build-id ruby -- PROTOCOL=MESI_Two_Level
./run --arch aarch64 --emulator -gem5 --gem5-build-id ruby -- --ruby</pre>
</div>
</div>
<div class="paragraph">
<p>and during build we see a humongous line of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[   SLICC] src/mem/protocol/MESI_Two_Level.slicc -&gt; ARM/mem/protocol/AccessPermission.cc, ARM/mem/protocol/AccessPermission.hh, ...</pre>
</div>
</div>
<div class="paragraph">
<p>which shows that dozens of C++ files are being generated from Ruby SLICC.</p>
</div>
<div class="paragraph">
<p>The relevant Ruby source files live in the source tree under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/mem/protocol/MESI_Two_Level*</pre>
</div>
</div>
<div class="paragraph">
<p>We already pass the <code>SLICC_HTML</code> flag by default to the build, which generates an HTML summary of each memory protocol under (TODO broken: <a href="https://gem5.atlassian.net/browse/GEM5-357" class="bare">https://gem5.atlassian.net/browse/GEM5-357</a>):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>xdg-open "$(./getvar --arch aarch64 --gem5-build-id ruby gem5_build_build_dir)/ARM/mem/protocol/html/index.html"</pre>
</div>
</div>
<div class="paragraph">
<p>A minimized ruby config which was not merged upstream can be found for study at: <a href="https://gem5-review.googlesource.com/c/public/gem5/+/13599/1" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/13599/1</a></p>
</div>
<div class="paragraph">
<p>One easy way to see that Ruby is being used without understanding it in detail is to <a href="#gem5-tracing">enable some logging</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-worktree master \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --static \
  --trace ExecAll,FmtFlag,Ruby,XBar \
  -- \
  --ruby \
;
cat "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>Then:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>when the <code>--ruby</code> flag is given, we see a gazillion Ruby related messages prefixed e.g. by <code>RubyPort:</code>.</p>
<div class="paragraph">
<p>We also observe from <code>ExecEnable</code> lines that instruction timing is not simple anymore, so the memory system must have latencies</p>
</div>
</li>
<li>
<p>without <code>--ruby</code>, we instead see <code>XBar</code> (Coherent Crossbar) related messages such as <code>CoherentXBar:</code>, which I believe is the more precise name for the memory model that the classic memory system uses: <a href="#gem5-crossbar-interconnect">gem5 crossbar interconnect</a>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Certain features may not work in Ruby. For example, <a href="#gem5-checkpoint">gem5 checkpoint</a> creation is only possible in Ruby protocols that support flush, which is the case for <code>PROTOCOL=MOESI_hammer</code> but not <code>PROTOCOL=MESI_Three_Level</code>: <a href="https://www.mail-archive.com/gem5-users@gem5.org/msg17418.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg17418.html</a></p>
</div>
<div class="paragraph">
<p>Tested in gem5 d7d9bc240615625141cd6feddbadd392457e49eb.</p>
</div>
<div class="sect4">
<h5 id="gem5-ruby-mi-example-protocol"><a class="anchor" href="#gem5-ruby-mi-example-protocol"></a><a class="link" href="#gem5-ruby-mi-example-protocol">24.16.6.1. gem5 Ruby MI_example protocol</a></h5>
<div class="paragraph">
<p>This is the simplest of all protocols, and therefore the first one you should study to learn how Ruby works.</p>
</div>
<div class="paragraph">
<p>To study it, we can take an approach similar to what was done at: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a>.</p>
</div>
<div class="paragraph">
<p>Our full command line will be something like</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --arch aarch64 --gem5-build-id MI_example
./run \
  --arch aarch64 \
  --cli-args '2 100' \
  --cpus 3 \
  --emulator gem5 \
  --userland userland/cpp/atomic/aarch64_add.cpp \
  --gem5-build-id MI_example \
  -- \
  --ruby \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which produces a <a href="#gem5-config-dot"><code>config.dot.svg</code></a> like the following by with 3 CPUs instead of 2:</p>
</div>
<div id="config-dot-svg-timingsimplecpu-caches-3-cpus-ruby" class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/media/master/gem5_config_TimingSimpleCPU_3_CPUs_MI_example_b1623cb2087873f64197e503ab8894b5e4d4c7b4.svg?sanitize=true" alt="gem5 config TimingSimpleCPU 3 CPUs MI example b1623cb2087873f64197e503ab8894b5e4d4c7b4" height="600">
</div>
<div class="title">Figure 2. <code>config.dot.svg</code> for a system with three TimingSimpleCPU CPUs with the Ruby <code>MI_example</code> protocol.</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-crossbar-interconnect"><a class="anchor" href="#gem5-crossbar-interconnect"></a><a class="link" href="#gem5-crossbar-interconnect">24.16.6.2. gem5 crossbar interconnect</a></h5>
<div class="paragraph">
<p>Crossbar or <code>XBar</code> in the code, is the default <a href="#cache-coherence">CPU interconnect</a> that gets used by <code>fs.py</code> if <a href="#gem5-ruby-build"><code>--ruby</code></a> is not given.</p>
</div>
<div class="paragraph">
<p>It presumably implements a crossbar switch along the lines of: <a href="https://en.wikipedia.org/wiki/Crossbar_switch" class="bare">https://en.wikipedia.org/wiki/Crossbar_switch</a></p>
</div>
<div class="paragraph">
<p>This is the best introductory example analysis we have so far: <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a>. It contains more or less the most minimal example in which something interesting can be observed: multiple cores fighting over a single data memory variable.</p>
</div>
<div class="paragraph">
<p>Long story short: the interconnect contains the snoop mechanism, and it forwards packets coming form caches of a CPU to the caches of other CPUs in which the block is present.</p>
</div>
<div class="paragraph">
<p>It is therefore the heart of the <a href="#cache-coherence">Cache coherence</a> mechanism, as it informs other caches of bus transactions they need to know about.</p>
</div>
<div class="paragraph">
<p>TODO: describe it in more detail. It appears to be a very simple mechanism.</p>
</div>
<div class="paragraph">
<p>Under <code>src/mem/</code> we see that there is both a coherent and a non-coherent XBar.</p>
</div>
<div class="paragraph">
<p>In <code>se.py</code> it is set at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if options.ruby:
    ...
else:
    MemClass = Simulation.setMemClass(options)
    system.membus = SystemXBar()</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>SystemXBar</code> is defined at <code>src/mem/XBar.py</code> with a nice comment:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># One of the key coherent crossbar instances is the system
# interconnect, tying together the CPU clusters, GPUs, and any I/O
# coherent masters, and DRAM controllers.
class SystemXBar(CoherentXBar):</pre>
</div>
</div>
<div class="paragraph">
<p>Tested in gem5 12c917de54145d2d50260035ba7fa614e25317a3.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-python-3-build"><a class="anchor" href="#gem5-python-3-build"></a><a class="link" href="#gem5-python-3-build">24.16.7. gem5 Python 3 build</a></h4>
<div class="paragraph">
<p>Python 3 support was mostly added in 2019 Q3 at arounda347a1a68b8a6e370334be3a1d2d66675891e0f1 but remained buggy for some time afterwards.</p>
</div>
<div class="paragraph">
<p>In an Ubuntu 18.04 host where <code>python</code> is <code>python2</code> by default, build with Python 3 instead with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --gem5-build-id python3 -- PYTHON_CONFIG=python3-config</pre>
</div>
</div>
<div class="paragraph">
<p>Python 3 is then automatically used when running if you use that build.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-cpu-types"><a class="anchor" href="#gem5-cpu-types"></a><a class="link" href="#gem5-cpu-types">24.17. gem5 CPU types</a></h3>
<div class="paragraph">
<p>gem5 has a few in tree CPU models for different purposes.</p>
</div>
<div class="paragraph">
<p>In fs.py and se.py, those are selectable with the <code>--cpu-type</code> option.</p>
</div>
<div class="paragraph">
<p>The information to make highly accurate models isn&#8217;t generally public for non-free CPUs, so either you must either rely vendor provided models or on experiments/reverse engineering.</p>
</div>
<div class="paragraph">
<p>There is no simple answer for "what is the best CPU", in theory you have to understand each model and decide which one is closer your target system.</p>
</div>
<div class="paragraph">
<p>Whenever possible, stick to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>vendor provide ones obviously, e.g. ARM Holdings models of ARM cores, unless there is good reason not to, as they are the most likely to be accurate</p>
</li>
<li>
<p>newer models instead of older models</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Both of those can be checked with <code>git log</code> and <code>git blame</code>.</p>
</div>
<div class="paragraph">
<p>All CPU types inherit from the <code>BaseCPU</code> class, and looking at the class hierarchy in <a href="#gem5-eclipse-configuration">Eclipse</a> gives a good overview of what we have:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>BaseCPU</code></p>
<div class="ulist">
<ul>
<li>
<p><code>BaseKvmCPU</code></p>
</li>
<li>
<p><code>BaseSimpleCPU</code>: <a href="#gem5-basesimplecpu">gem5 <code>BaseSimpleCPU</code></a></p>
<div class="ulist">
<ul>
<li>
<p><code>AtomicSimpleCPU</code></p>
</li>
<li>
<p><code>TimingSimpleCPU</code></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>MinorO3CPU</code>: <a href="#gem5-minorcpu">gem5 MinorCPU</a></p>
</li>
<li>
<p><code>BaseO3CPU</code></p>
<div class="ulist">
<ul>
<li>
<p><code>FullO3CPU</code></p>
<div class="ulist">
<ul>
<li>
<p><code>DerivO3CPU : public FullO3CPU&lt;O3CPUImpl&gt;</code>: <a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>From this we see that there are basically only 4 C++ CPU models in gem5: Atomic, Timing, Minor and O3. All others are basically parametrizations of those base types.</p>
</div>
<div class="sect3">
<h4 id="list-of-gem5-cpu-types"><a class="anchor" href="#list-of-gem5-cpu-types"></a><a class="link" href="#list-of-gem5-cpu-types">24.17.1. List of gem5 CPU types</a></h4>
<div class="sect4">
<h5 id="gem5-basesimplecpu"><a class="anchor" href="#gem5-basesimplecpu"></a><a class="link" href="#gem5-basesimplecpu">24.17.1.1. gem5 <code>BaseSimpleCPU</code></a></h5>
<div class="paragraph">
<p>Simple abstract CPU without a pipeline.</p>
</div>
<div class="paragraph">
<p>They are therefore completely unrealistic. But they also run much faster. <a href="#gem5-kvm">KVM CPUs</a> are an alternative way of fast forwarding boot when they work.</p>
</div>
<div class="paragraph">
<p>Implementations:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-atomicsimplecpu">gem5 <code>AtomicSimpleCPU</code></a></p>
</li>
<li>
<p><a href="#gem5-timingsimplecpu">gem5 <code>TimingSimpleCPU</code></a></p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="gem5-atomicsimplecpu"><a class="anchor" href="#gem5-atomicsimplecpu"></a><a class="link" href="#gem5-atomicsimplecpu">24.17.1.1.1. gem5 <code>AtomicSimpleCPU</code></a></h6>
<div class="paragraph">
<p><code>AtomicSimpleCPU</code>: the default one. Memory accesses happen instantaneously. The fastest simulation except for KVM, but not realistic at all.</p>
</div>
<div class="paragraph">
<p>Useful to <a href="#gem5-restore-checkpoint-with-a-different-cpu">boot Linux fast and then checkpoint and switch to a more detailed CPU</a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-timingsimplecpu"><a class="anchor" href="#gem5-timingsimplecpu"></a><a class="link" href="#gem5-timingsimplecpu">24.17.1.1.2. gem5 <code>TimingSimpleCPU</code></a></h6>
<div class="paragraph">
<p><code>TimingSimpleCPU</code>: memory accesses are realistic, but the CPU has no pipeline. The simulation is faster than detailed models, but slower than <code>AtomicSimpleCPU</code>.</p>
</div>
<div class="paragraph">
<p>To fully understand <code>TimingSimpleCPU</code>, see: <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a>.</p>
</div>
<div class="paragraph">
<p>Without caches, the CPU just stalls all the time waiting for memory requests for every advance of the PC or memory read from a instruction!</p>
</div>
<div class="paragraph">
<p>Caches do make a difference here of course, and lead to much faster memory return times.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-minorcpu"><a class="anchor" href="#gem5-minorcpu"></a><a class="link" href="#gem5-minorcpu">24.17.1.2. gem5 MinorCPU</a></h5>
<div class="paragraph">
<p>Generic <a href="#out-of-order-execution">in-order</a> <a href="#superscalar-processor">superscalar</a> core.</p>
</div>
<div class="paragraph">
<p>Its C++ implementation that can be parametrized to more closely match real cores.</p>
</div>
<div class="paragraph">
<p>Note that since gem5 is highly parametrizable, the parametrization could even change which instructions a CPU can execute by altering its available <a href="#gem5-functional-units">functional units</a>, which are used to model performance.</p>
</div>
<div class="paragraph">
<p>For example, <code>MinorCPU</code> allows all implemented instructions, including <a href="#arm-sve">ARM SVE</a> instructions, but a derived class modelling, say, an <a href="https://en.wikipedia.org/wiki/ARM_Cortex-A7">ARM Cortex A7 core</a>, might not, since SVE is a newer feature and the A7 core does not have SVE.</p>
</div>
<div class="paragraph">
<p>The weird name "Minor" stands for "M (TODO what is M) IN ONder".</p>
</div>
<div class="paragraph">
<p>Its 4 stage pipeline is described at the "MinorCPU" section of <a href="#gem5-arm-rsk">gem5 ARM RSK</a>.</p>
</div>
<div class="paragraph">
<p>A commented execution example can be seen at: <a href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis">gem5 event queue MinorCPU syscall emulation freestanding example analysis</a>.</p>
</div>
<div class="paragraph">
<p>There is also an in-tree doxygen at: <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/doc/inside-minor.doxygen"><code>src/doc/inside-minor.doxygen</code></a> and rendered at: <a href="http://pages.cs.wisc.edu/~swilson/gem5-docs/minor.html" class="bare">http://pages.cs.wisc.edu/~swilson/gem5-docs/minor.html</a></p>
</div>
<div class="paragraph">
<p>As of 2019, in-order cores are mostly present in low power/cost contexts, for example little cores of <a href="https://en.wikipedia.org/wiki/ARM_big.LITTLE">ARM bigLITTLE</a>.</p>
</div>
<div class="paragraph">
<p>The following models extend the <code>MinorCPU</code> class by parametrization to make it match existing CPUs more closely:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>HPI</code>: derived from <code>MinorCPU</code>.</p>
<div class="paragraph">
<p>Created by Ashkan Tousi in 2017 while working at ARM.</p>
</div>
<div class="paragraph">
<p>According to <a href="#gem5-arm-rsk">gem5 ARM RSK</a>:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The HPI CPU timing model is tuned to be representative of a modern in-order Armv8-A implementation.</p>
</div>
</blockquote>
</div>
</li>
<li>
<p><code>ex5_LITTLE</code>: derived from <code>MinorCPU</code>. Description reads:</p>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>ex5 LITTLE core (based on the ARM Cortex-A7)</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Implemented by Pierre-Yves Péneau from LIRMM, which is a research lab in Montpellier, France, in 2017.</p>
</div>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="gem5-derivo3cpu"><a class="anchor" href="#gem5-derivo3cpu"></a><a class="link" href="#gem5-derivo3cpu">24.17.1.3. gem5 <code>DerivO3CPU</code></a></h5>
<div class="paragraph">
<p>Generic <a href="#out-of-order-execution">out-of-order core</a>. "O3" Stands for "Out Of Order"!</p>
</div>
<div class="paragraph">
<p>Basic documentation on the old gem5 wiki: <a href="http://www.m5sim.org/O3CPU" class="bare">http://www.m5sim.org/O3CPU</a></p>
</div>
<div class="paragraph">
<p>Analogous to <a href="#gem5-minorcpu">MinorCPU</a>, but modelling an out of order core instead of in order.</p>
</div>
<div class="paragraph">
<p>A commented execution example can be seen at: <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis</a>.</p>
</div>
<div class="paragraph">
<p>The default <a href="#execution-unit">functional units</a> are described at: <a href="#gem5-derivo3cpu-default-functional-units">gem5 DerivO3CPU default functional units</a>. All default widths are set to 8 instructions, from the <a href="#gem5-config-ini"><code>config.ini</code></a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu]
type=DerivO3CPU
commitWidth=8
decodeWidth=8
dispatchWidth=8
fetchWidth=8
issueWidth=8
renameWidth=8
squashWidth=8
wbWidth=8</pre>
</div>
</div>
<div class="paragraph">
<p>This can be observed for example at: <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless</a>.</p>
</div>
<div class="paragraph">
<p>Existing parametrizations:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ex5_big</code>: big corresponding to <code>ex5_LITTLE</code>, by same author at same time. It description reads:</p>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>ex5 big core (based on the ARM Cortex-A15)</p>
</div>
</blockquote>
</div>
</li>
<li>
<p><code>O3_ARM_v7a</code>: implemented by Ronald Dreslinski from the <a href="https://en.wikipedia.org/wiki/University_of_Michigan">University of Michigan</a> in 2012</p>
<div class="paragraph">
<p>Not sure why it has v7a in the name, since I believe the CPUs are just the microarchitectural implementation of any ISA, and the v8 hello world did run.</p>
</div>
<div class="paragraph">
<p>The CLI option is named slightly differently as: <code>--cpu-type O3_ARM_v7a_3</code>.</p>
</div>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="gem5-derivo3cpu-pipeline-stages"><a class="anchor" href="#gem5-derivo3cpu-pipeline-stages"></a><a class="link" href="#gem5-derivo3cpu-pipeline-stages">24.17.1.3.1. gem5 <code>DerivO3CPU</code> pipeline stages</a></h6>
<div class="ulist">
<ul>
<li>
<p>fetch: besides obviously fetching the instruction, this is also where branch prediction runs. Presumably because you need to branch predict before deciding what to fetch next.</p>
</li>
<li>
<p>retire: the instruction is completely and totally done with.</p>
<div class="paragraph">
<p>Mispeculated instructions never reach this stage as can be seen at: <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative</a>.</p>
</div>
<div class="paragraph">
<p>The <code>ExecAll</code> happens at this time as well. And therefore <code>ExecAll</code> does not happen for mispeculated instructions.</p>
</div>
</li>
</ul>
</div>
</div>
<div class="sect5">
<h6 id="gem5-util-o3-pipeview-py-o3-pipeline-viewer"><a class="anchor" href="#gem5-util-o3-pipeview-py-o3-pipeline-viewer"></a><a class="link" href="#gem5-util-o3-pipeview-py-o3-pipeline-viewer">24.17.1.3.2. gem5 util/o3-pipeview.py O3 pipeline viewer</a></h6>
<div class="paragraph">
<p>Mentioned at: <a href="http://www.m5sim.org/Visualization" class="bare">http://www.m5sim.org/Visualization</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace O3PipeView \
  --trace-stdout \
  -- \
  --cpu-type DerivO3CPU \
  --caches \
;
"$(./getvar gem5_source_dir)/util/o3-pipeview.py" -c 500 -o o3pipeview.tmp.log --color "$(./getvar --arch aarch64 trace_txt_file)"
less -R o3pipeview.tmp.log</pre>
</div>
</div>
<div class="paragraph">
<p>Or without color:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>"$(./getvar gem5_source_dir)/util/o3-pipeview.py" -c 500 -o o3pipeview.tmp.log "$(./getvar --arch aarch64 trace_txt_file)"
less o3pipeview.tmp.log</pre>
</div>
</div>
<div class="paragraph">
<p>A sample output for this can be seen at: <a href="#hazardless-o3-pipeline">[hazardless-o3-pipeline]</a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-konata-o3-pipeline-viewer"><a class="anchor" href="#gem5-konata-o3-pipeline-viewer"></a><a class="link" href="#gem5-konata-o3-pipeline-viewer">24.17.1.3.3. gem5 Konata O3 pipeline viewer</a></h6>
<div class="paragraph">
<p><a href="https://github.com/shioyadan/Konata" class="bare">https://github.com/shioyadan/Konata</a></p>
</div>
<div class="paragraph">
<p><a href="http://learning.gem5.org/tutorial/presentations/vis-o3-gem5.pdf" class="bare">http://learning.gem5.org/tutorial/presentations/vis-o3-gem5.pdf</a></p>
</div>
<div class="paragraph">
<p>Appears to be browser based, so you can zoom in and out, rather than the forced wrapping as for <a href="#gem5-util-o3-pipeview-py-o3-pipeline-viewer">gem5 util/o3-pipeview.py O3 pipeline viewer</a>.</p>
</div>
<div class="paragraph">
<p>Uses the same data source as <code>util/o3-pipeview.py</code>.</p>
</div>
<div class="paragraph">
<p><a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_gain</a> shows how the text-based visualization can get problematic due to stalls requiring wraparounds.</p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-arm-rsk"><a class="anchor" href="#gem5-arm-rsk"></a><a class="link" href="#gem5-arm-rsk">24.17.2. gem5 ARM RSK</a></h4>
<div class="paragraph">
<p><a href="https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/gem5_rsk.pdf" class="bare">https://github.com/arm-university/arm-gem5-rsk/blob/aa3b51b175a0f3b6e75c9c856092ae0c8f2a7cdc/gem5_rsk.pdf</a></p>
</div>
<div class="paragraph">
<p>Dated 2017, it contains a good overview of gem5 CPUs.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-arm-platforms"><a class="anchor" href="#gem5-arm-platforms"></a><a class="link" href="#gem5-arm-platforms">24.18. gem5 ARM platforms</a></h3>
<div class="paragraph">
<p>The gem5 platform is selectable with the <code>--machine</code> option, which is named after the analogous QEMU <code>-machine</code> option, and which sets the <code>--machine-type</code>.</p>
</div>
<div class="paragraph">
<p>Each platform represents a different system with different devices, memory and interrupt setup.</p>
</div>
<div class="paragraph">
<p>TODO: describe the main characteristics of each platform, as of gem5 5e83d703522a71ec4f3eb61a01acd8c53f6f3860:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>VExpress_GEM5_V1</code>: good sane base platform</p>
</li>
<li>
<p><code>VExpress_GEM5_V1_DPU</code>: <code>VExpress_GEM5_V1</code> with DP650 instead of HDLCD, selected automatically by <code>./run --dp650</code>, see also: <a href="#gem5-graphic-mode-dp650">gem5 graphic mode DP650</a></p>
</li>
<li>
<p><code>VExpress_GEM5_V2</code>: VExpress_GEM5_V1 with GICv3, uses a different bootloader <code>arm/aarch64_bootloader/boot_emm_v2.arm64</code> TODO is it because of GICv3?</p>
</li>
<li>
<p>anything that does not start with: <code>VExpress_GEM5_</code>: old and bad, don&#8217;t use them</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="gem5-upstream-images"><a class="anchor" href="#gem5-upstream-images"></a><a class="link" href="#gem5-upstream-images">24.19. gem5 upstream images</a></h3>
<div class="paragraph">
<p>Present at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://www.gem5.org/dist/current/arm/" class="bare">http://www.gem5.org/dist/current/arm/</a></p>
</li>
<li>
<p><a href="http://www.gem5.org/dist/current/x86/" class="bare">http://www.gem5.org/dist/current/x86/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Depending on which archive you download from there, you can find some of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Ubuntu based images</p>
</li>
<li>
<p>precompiled Linux kernels, with the <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a> for arm</p>
</li>
<li>
<p>precompiled <a href="#gem5-bootloaders">gem5 bootloaders</a> for ISAs that have them, e.g. ARM</p>
</li>
<li>
<p>precompiled DTBs if you don&#8217;t want to use autogeneration for some crazy reason</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Some of those images are also used on the <a href="#gem5-unit-tests">gem5 unit tests</a> continuous integration.</p>
</div>
<div class="paragraph">
<p>Could be used as an alternative to this repository. But why would you do that? :-)</p>
</div>
<div class="paragraph">
<p>E.g. to use a precompiled ARM kernel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir aarch-system-201901106
cd aarch-system-201901106
wget http://dist.gem5.org/dist/current/arm/aarch-system-201901106.tar.bz2
tar xvf aarch-system-201901106.tar.bz2
cd ..
./run --arch aarch64 --emulator gem5 --linux-exec aarch-system-201901106/binaries/vmlinux.arm64</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-bootloaders"><a class="anchor" href="#gem5-bootloaders"></a><a class="link" href="#gem5-bootloaders">24.20. gem5 bootloaders</a></h3>
<div class="paragraph">
<p>Certain ISAs like ARM have bootloaders that are automatically run before the main image to setup basic system state.</p>
</div>
<div class="paragraph">
<p>We cross compile those bootloaders from source automatically during <code>./build-gem5</code>.</p>
</div>
<div class="paragraph">
<p>As of gem5 bcf041f257623e5c9e77d35b7531bae59edc0423, the source code of the bootloaderes can be found under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system/arm/</pre>
</div>
</div>
<div class="paragraph">
<p>and their selection can be seen under: <code>src/dev/arm/RealView.py</code>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    def setupBootLoader(self, cur_sys, loc):
        if not cur_sys.boot_loader:
            cur_sys.boot_loader = [ loc('boot_emm.arm64'), loc('boot_emm.arm') ]</pre>
</div>
</div>
<div class="paragraph">
<p>The bootloader basically just sets up a bit of CPU state and jumps to the kernel entry point.</p>
</div>
<div class="paragraph">
<p>In aarch64 at least, CPUs other than CPU0 are also started up briefly, run some initialization, and are made wait on a WFE. This can be seen easily by booting a multicore Linux kernel run with <a href="#gem5-execall-trace-format">gem5 ExecAll trace format</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="gem5-memory-system"><a class="anchor" href="#gem5-memory-system"></a><a class="link" href="#gem5-memory-system">24.21. gem5 memory system</a></h3>
<div class="paragraph">
<p>Parent section: <a href="#gem5-internals">gem5 internals</a>.</p>
</div>
<div class="sect3">
<h4 id="gem5-port-system"><a class="anchor" href="#gem5-port-system"></a><a class="link" href="#gem5-port-system">24.21.1. gem5 port system</a></h4>
<div class="paragraph">
<p>The gem5 memory system is connected in a very flexible way through the port system.</p>
</div>
<div class="paragraph">
<p>This system exists to allow seamlessly connecting any combination of CPU, caches, interconnects, DRAM and peripherals.</p>
</div>
<div class="paragraph">
<p>A <a href="#gem5-packet"><code>Packet</code></a> is the basic information unit that gets sent across ports.</p>
</div>
<div class="sect4">
<h5 id="gem5-functional-vs-atomic-vs-timing-memory-requests"><a class="anchor" href="#gem5-functional-vs-atomic-vs-timing-memory-requests"></a><a class="link" href="#gem5-functional-vs-atomic-vs-timing-memory-requests">24.21.1.1. gem5 functional vs atomic vs timing memory requests</a></h5>
<div class="paragraph">
<p>gem5 memory requests can be classified in the following broad categories:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>functional: get the value magically, do not update caches, see also: <a href="#gem5-functional-requests">gem5 functional requests</a></p>
</li>
<li>
<p>atomic: get the value now without making a <a href="#gem5-event-queue">separate event</a>, but do not update caches. Cannot work in <a href="#gem5-ruby-build">Ruby</a> due to fundamental limitations, mentioned in passing at: <a href="https://gem5.atlassian.net/browse/GEM5-676" class="bare">https://gem5.atlassian.net/browse/GEM5-676</a></p>
</li>
<li>
<p>timing: get the value simulating delays and updating caches</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This trichotomy can be notably seen in the definition of the <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/port.hh#L75">MasterPort class</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class MasterPort : public Port, public AtomicRequestProtocol,
    public TimingRequestProtocol, public FunctionalRequestProtocol</pre>
</div>
</div>
<div class="paragraph">
<p>and the base classes are defined under <code>src/mem/protocol/</code>.</p>
</div>
<div class="paragraph">
<p>Then, by reading the rest of the class, we see that the send methods are all boring, and just forward to some polymorphic receiver that does the actual interesting activity:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Tick
    sendAtomicSnoop(PacketPtr pkt)
    {
        return AtomicResponseProtocol::sendSnoop(_masterPort, pkt);
    }

    Tick
    AtomicResponseProtocol::sendSnoop(AtomicRequestProtocol *peer, PacketPtr pkt)
    {
        assert(pkt-&gt;isRequest());
        return peer-&gt;recvAtomicSnoop(pkt);
    }</pre>
</div>
</div>
<div class="paragraph">
<p>The receive methods are therefore the interesting ones, and must be overridden on derived classes if they ever expect to receive such requests:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Tick
    recvAtomicSnoop(PacketPtr pkt) override
    {
        panic("%s was not expecting an atomic snoop request\n", name());
        return 0;
    }

    void
    recvFunctionalSnoop(PacketPtr pkt) override
    {
        panic("%s was not expecting a functional snoop request\n", name());
    }

    void
    recvTimingSnoopReq(PacketPtr pkt) override
    {
        panic("%s was not expecting a timing snoop request.\n", name());
    }</pre>
</div>
</div>
<div class="paragraph">
<p>One question that comes up now is: but why do CPUs need to care about <a href="#cache-coherence">snoop requests</a>?</p>
</div>
<div class="paragraph">
<p>And one big answer is: to be able to implement LLSC atomicity as mentioned at: <a href="#arm-ldxr-and-stxr-instructions">ARM LDXR and STXR instructions</a>, since when other cores update memory, they could invalidate the lock of the current core.</p>
</div>
<div class="paragraph">
<p>Then, as you might expect, we can see that for example <code>AtomicSimpleCPU</code> does not override <code>recvTimingSnoopReq</code>.</p>
</div>
<div class="paragraph">
<p>Now let see which requests are generated by ordinary <a href="#arm-ldr-instruction">ARM LDR instruction</a>. We run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --debug-vm \
  --emulator gem5 \
  --gem5-build-type debug \
  --useland userland/arch/aarch64/freestanding/linux/hello.S \</pre>
</div>
</div>
<div class="paragraph">
<p>and then break at the methods of the LDR class <code>LDRXL64_LIT</code>: <a href="#gem5-execute-vs-initiateacc-vs-completeacc">gem5 <code>execute</code> vs <code>initiateAcc</code> vs <code>completeAcc</code></a>.</p>
</div>
<div class="paragraph">
<p>Before starting, we of course guess that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>AtomicSimpleCPU</code> will be making atomic accesses from <code>execute</code></p>
</li>
<li>
<p><code>TimingSimpleCPU</code> will be making timing accesses from <code>initiateAcc</code>, which must generate the event which leads to <code>completeAcc</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so let&#8217;s confirm it.</p>
</div>
<div class="paragraph">
<p>We break on <code>ArmISAInst::LDRXL64_LIT::execute</code> which is what <code>AtomicSimpleCPU</code> uses, and that leads as expected to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>MasterPort::sendAtomic
AtomicSimpleCPU::sendPacket
AtomicSimpleCPU::readMem
SimpleExecContext::readMem
readMemAtomic&lt;(ByteOrder)1, ExecContext, unsigned long&gt;
readMemAtomicLE&lt;ExecContext, unsigned long&gt;
ArmISAInst::LDRXL64_LIT::execute
AtomicSimpleCPU::tick</pre>
</div>
</div>
<div class="paragraph">
<p>Notably, <code>AtomicSimpleCPU::readMem</code> immediately translates the address, creates a packet, sends the atomic request, and gets the response back without any events.</p>
</div>
<div class="paragraph">
<p>And now if we do the same with <code>--cpu-type TimingSimpleCPU</code> and break at <code>ArmISAInst::LDRXL64_LIT::initiateAcc</code>, and then add another break for the next event schedule <code>b EventManager::schedule</code> (which we imagine is the memory read) we reach:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::addToReadQueue
DRAMCtrl::recvTimingReq
DRAMCtrl::MemoryPort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
CoherentXBar::recvTimingReq
CoherentXBar::CoherentXBarSlavePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::handleReadPacket
TimingSimpleCPU::sendData
TimingSimpleCPU::finishTranslation
DataTranslation&lt;TimingSimpleCPU*&gt;::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::initiateMemRead
SimpleExecContext::initiateMemRead
initiateMemRead&lt;ExecContext, unsigned long&gt;
ArmISAInst::LDRXL64_LIT::initiateAcc
TimingSimpleCPU::completeIfetch
TimingSimpleCPU::IcachePort::ITickEvent::process
EventQueue::serviceOne</pre>
</div>
</div>
<div class="paragraph">
<p>so as expected we have <code>TimingRequestProtocol::sendReq</code>.</p>
</div>
<div class="paragraph">
<p>Remember however that timing requests are a bit more complicated due to <a href="#arm-paging">paging</a>, since the page table walk can itself lead to further memory requests.</p>
</div>
<div class="paragraph">
<p>In this particular instance, the address being read with <code>ldr x2, =len</code> <a href="#arm-ldr-pseudo-instruction">ARM LDR pseudo-instruction</a> is likely placed just after the text section, and therefore the pagewalk is already in the TLB due to previous instruction fetches, and this is because the translation just finished immediately going through <code>TimingSimpleCPU::finishTranslation</code>, some key snippets are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>TLB::translateComplete(const RequestPtr &amp;req, ThreadContext *tc,
        Translation *translation, Mode mode, TLB::ArmTranslationType tranType,
        bool callFromS2)
{
    bool delay = false;
    Fault fault;
    if (FullSystem)
        fault = translateFs(req, tc, mode, translation, delay, true, tranType);
    else
        fault = translateSe(req, tc, mode, translation, delay, true);
    if (!delay)
        translation-&gt;finish(fault, req, tc, mode);
    else
        translation-&gt;markDelayed();</pre>
</div>
</div>
<div class="paragraph">
<p>and then <code>translateSe</code> does not use <code>delay</code> at all, so we learn that in syscall emulation, <code>delay</code> is always <code>false</code> and things progress immediately there. And then further down <code>TimingSimpleCPU::finishTranslation</code> does some more fault checking:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
TimingSimpleCPU::finishTranslation(WholeTranslationState *state)
{
    if (state-&gt;getFault() != NoFault) {
        translationFault(state-&gt;getFault());
    } else {
        if (!state-&gt;isSplit) {
            sendData(state-&gt;mainReq, state-&gt;data, state-&gt;res,
                     state-&gt;mode == BaseTLB::Read);</pre>
</div>
</div>
<div class="paragraph">
<p>Tested in gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.</p>
</div>
<div class="sect5">
<h6 id="gem5-functional-requests"><a class="anchor" href="#gem5-functional-requests"></a><a class="link" href="#gem5-functional-requests">24.21.1.1.1. gem5 functional requests</a></h6>
<div class="paragraph">
<p>As seen at <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">gem5 functional vs atomic vs timing memory requests</a>, functional requests are not used in common simulation, since the core must always go through caches.</p>
</div>
<div class="paragraph">
<p>Functional access are therefore only used for more magic simulation functionalities.</p>
</div>
<div class="paragraph">
<p>One such functionality, is the <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a> implementation of the <a href="#futex-system-call">futex system call</a> which is done at <code>futexFunc</code> in <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/sim/syscall_emul.hh#L394"><code>src/sim/sycall_emul.hh</code></a>.</p>
</div>
<div class="paragraph">
<p>As seen from <code>man futex</code>, the Linux kernel reads the value from an address that is given as the first argument of the call.</p>
</div>
<div class="paragraph">
<p>Therefore, here it makes sense for gem5 syscall implementation, which does not actually have a real kernel running, to just make a functional request and be done with it, since the impact of cache changes done by this read would be insignificant to the cost of an actual full context switch that would happen on a real syscall.</p>
</div>
<div class="paragraph">
<p>It is generally hard to implement functional requests for <a href="#gem5-ruby-build">Ruby</a> runs, because packets are flying through the memory system in a transient state, and there is no simple way of finding exactly which ones might have the latest version of the memory. See for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://gem5.atlassian.net/browse/GEM5-496" class="bare">https://gem5.atlassian.net/browse/GEM5-496</a></p>
</li>
<li>
<p><a href="https://gem5.atlassian.net/browse/GEM5-604" class="bare">https://gem5.atlassian.net/browse/GEM5-604</a></p>
</li>
<li>
<p><a href="https://gem5.atlassian.net/browse/GEM5-675" class="bare">https://gem5.atlassian.net/browse/GEM5-675</a></p>
</li>
<li>
<p><a href="https://gem5.atlassian.net/browse/GEM5-676" class="bare">https://gem5.atlassian.net/browse/GEM5-676</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The typical error message in that case is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fatal: Ruby functional read failed for address</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-packet-vs-request"><a class="anchor" href="#gem5-packet-vs-request"></a><a class="link" href="#gem5-packet-vs-request">24.21.2. gem5 <code>Packet</code> vs <code>Request</code></a></h4>
<div class="sect4">
<h5 id="gem5-packet"><a class="anchor" href="#gem5-packet"></a><a class="link" href="#gem5-packet">24.21.2.1. gem5 <code>Packet</code></a></h5>
<div class="paragraph">
<p><code>Packet</code> is what goes through <a href="#gem5-port-system">ports</a>: a single packet is sent out to the memory system, gets modified when it hits valid data, and then returns with the reply.</p>
</div>
<div class="paragraph">
<p><code>Packet</code> is what CPUs create and send to get memory values. E.g. on <a href="#gem5-atomicsimplecpu">gem5 <code>AtomicSimpleCPU</code></a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
AtomicSimpleCPU::tick()
{
    ...
    Packet ifetch_pkt = Packet(ifetch_req, MemCmd::ReadReq);
    ifetch_pkt.dataStatic(&amp;inst);

    icache_latency = sendPacket(icachePort, &amp;ifetch_pkt);

Tick
AtomicSimpleCPU::sendPacket(MasterPort &amp;port, const PacketPtr &amp;pkt)
{
    return port.sendAtomic(pkt);
}</pre>
</div>
</div>
<div class="paragraph">
<p>On <a href="#gem5-timingsimplecpu">TimingSimpleCPU</a>, we note that the packet is dynamically created unlike for the AtomicSimpleCPU, since it must exist across multiple <a href="#gem5-event-queue">events</a> which happen on separate function calls, unlike atomic memory which is done immediately in a single call:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
TimingSimpleCPU::sendFetch(const Fault &amp;fault, const RequestPtr &amp;req,
                           ThreadContext *tc)
{
    if (fault == NoFault) {
        DPRINTF(SimpleCPU, "Sending fetch for addr %#x(pa: %#x)\n",
                req-&gt;getVaddr(), req-&gt;getPaddr());
        ifetch_pkt = new Packet(req, MemCmd::ReadReq);
        ifetch_pkt-&gt;dataStatic(&amp;inst);
        DPRINTF(SimpleCPU, " -- pkt addr: %#x\n", ifetch_pkt-&gt;getAddr());

        if (!icachePort.sendTimingReq(ifetch_pkt)) {</pre>
</div>
</div>
<div class="paragraph">
<p>It must later delete the return packet that it gets later on, e.g. for the ifetch:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>TimingSimpleCPU::completeIfetch(PacketPtr pkt)
{
    if (pkt) {
        delete pkt;
    }</pre>
</div>
</div>
<div class="paragraph">
<p>The most important properties of a Packet are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>PacketDataPtr data;</code>: the data coming back from a reply packet or being sent via it</p>
</li>
<li>
<p><code>Addr addr;</code>: the physical address of the data. TODO comment says could be virtual too, when?</p>
<div class="literalblock">
<div class="content">
<pre>/// The address of the request.  This address could be virtual or
/// physical, depending on the system configuration.
Addr addr;</pre>
</div>
</div>
</li>
<li>
<p><code>Flags flags;</code>: flags describing properties of the <code>Packet</code></p>
</li>
<li>
<p><code>MemCmd cmd;</code>: see <a href="#gem5-memcmd">gem5 <code>MemCmd</code></a></p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="gem5-memcmd"><a class="anchor" href="#gem5-memcmd"></a><a class="link" href="#gem5-memcmd">24.21.2.1.1. gem5 <code>MemCmd</code></a></h6>
<div class="paragraph">
<p>Each <a href="#gem5-packet">gem5 <code>Packet</code></a> contains a <code>MemCmd</code></p>
</div>
<div class="paragraph">
<p>The <code>MemCmd</code> is basically an enumeration of possible commands, stuff like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>enum Command
{
    InvalidCmd,
    ReadReq,
    ReadResp,</pre>
</div>
</div>
<div class="paragraph">
<p>Each command has a fixed number of attributes defined in the static array:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static const CommandInfo commandInfo[];</pre>
</div>
</div>
<div class="paragraph">
<p>which gets initialized in the .cc file in the same order as the Command enum.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>const MemCmd::CommandInfo
MemCmd::commandInfo[] =
{
    /* InvalidCmd */
    { 0, InvalidCmd, "InvalidCmd" },
    /* ReadReq - Read issued by a non-caching agent such as a CPU or
     * device, with no restrictions on alignment. */
    { SET3(IsRead, IsRequest, NeedsResponse), ReadResp, "ReadReq" },
    /* ReadResp */
    { SET3(IsRead, IsResponse, HasData), InvalidCmd, "ReadResp" },</pre>
</div>
</div>
<div class="paragraph">
<p>From this we see for example that both <code>ReadReq</code> and <code>ReadResp</code> are marked with the <code>IsRead</code> attribute.</p>
</div>
<div class="paragraph">
<p>The second field of this array also specifies the corresponding reply of each request. E.g. the reply of a <code>ReadReq</code> is a <code>ReadResp</code>. <code>InvalidCmd</code> is just a placeholders for requests that are already replies.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>struct CommandInfo
{
    /// Set of attribute flags.
    const std::bitset&lt;NUM_COMMAND_ATTRIBUTES&gt; attributes;
    /// Corresponding response for requests; InvalidCmd if no
    /// response is applicable.
    const Command response;
    /// String representation (for printing)
    const std::string str;
};</pre>
</div>
</div>
<div class="paragraph">
<p>Some important commands include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ReadReq</code>: what the CPU sends out to its cache, see also: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></p>
</li>
<li>
<p><code>ReadSharedReq</code>: what dcache of the CPU sends forward to the <a href="#gem5-crossbar-interconnect">gem5 crossbar interconnect</a> after a <code>ReadReq</code>, see also: see also: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></p>
</li>
<li>
<p><code>ReadResp</code>: response to a <code>ReadReq</code>. Can come from either DRAM or another cache that has the data. On <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a> we see that a new packet is created.</p>
</li>
<li>
<p><code>WriteReq</code>: what the CPU sends out to its cache, see also: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></p>
</li>
<li>
<p><code>UpgradeReq</code>: what dcache of CPU sends forward after a <code>WriteReq</code></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-request"><a class="anchor" href="#gem5-request"></a><a class="link" href="#gem5-request">24.21.2.2. gem5 <code>Request</code></a></h5>
<div class="paragraph">
<p>One good way to think about <code>Request</code> vs <code>Packet</code> could be "it is what the <a href="#gem5-instruction-definitions">instruction definitions</a> see", a bit like <code>ExecContext</code> vs <code>ThreadContext</code>.</p>
</div>
<div class="paragraph">
<p><code>Request</code> is passed to the constructor of <code>Packet</code>, and <code>Packet</code> keeps a reference to it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Packet(const RequestPtr &amp;_req, MemCmd _cmd)
        :  cmd(_cmd), id((PacketId)_req.get()), req(_req),
           data(nullptr), addr(0), _isSecure(false), size(0),
           _qosValue(0), headerDelay(0), snoopDelay(0),
           payloadDelay(0), senderState(NULL)
    {
        if (req-&gt;hasPaddr()) {
            addr = req-&gt;getPaddr();
            flags.set(VALID_ADDR);
            _isSecure = req-&gt;isSecure();
        }
        if (req-&gt;hasSize()) {
            size = req-&gt;getSize();
            flags.set(VALID_SIZE);
        }
    }</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>RequestPtr</code> is defined as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>typedef std::shared_ptr&lt;Request&gt; RequestPtr;</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that shared pointers to requests are basically passed around.</p>
</div>
<div class="paragraph">
<p>Some key fields include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>_paddr</code>:</p>
<div class="literalblock">
<div class="content">
<pre>/**
    * The physical address of the request. Valid only if validPaddr
    * is set.
    */
Addr _paddr = 0;</pre>
</div>
</div>
</li>
<li>
<p><code>_vaddr</code>:</p>
<div class="literalblock">
<div class="content">
<pre>/** The virtual address of the request. */
Addr _vaddr = MaxAddr;</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="gem5-request-in-atomicsimplecpu"><a class="anchor" href="#gem5-request-in-atomicsimplecpu"></a><a class="link" href="#gem5-request-in-atomicsimplecpu">24.21.2.2.1. gem5 <code>Request</code> in <code>AtomicSimpleCPU</code></a></h6>
<div class="paragraph">
<p>In <code>AtomicSimpleCPU</code>, a single packet of each type is kept for the entire CPU, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>RequestPtr ifetch_req;</pre>
</div>
</div>
<div class="paragraph">
<p>and it gets created at construction time:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>AtomicSimpleCPU::AtomicSimpleCPU(AtomicSimpleCPUParams *p)
{
    ifetch_req = std::make_shared&lt;Request&gt;();</pre>
</div>
</div>
<div class="paragraph">
<p>and then it gets modified for each request:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>setupFetchRequest(ifetch_req);</pre>
</div>
</div>
<div class="paragraph">
<p>which does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>req-&gt;setVirt(fetchPC, sizeof(MachInst), Request::INST_FETCH,
                instMasterId(), instAddr);</pre>
</div>
</div>
<div class="paragraph">
<p>Virtual to physical address translation done by the CPU stores the physical address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fault = thread-&gt;dtb-&gt;translateAtomic(req, thread-&gt;getTC(),
                                        BaseTLB::Read);</pre>
</div>
</div>
<div class="paragraph">
<p>which eventually calls e.g. on fs with MMU enabled:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault
TLB::translateMmuOn(ThreadContext* tc, const RequestPtr &amp;req, Mode mode,
                    Translation *translation, bool &amp;delay, bool timing,
                    bool functional, Addr vaddr,
                    ArmFault::TranMethod tranMethod)
{
    req-&gt;setPaddr(pa);</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="gem5-request-in-timingsimplecpu"><a class="anchor" href="#gem5-request-in-timingsimplecpu"></a><a class="link" href="#gem5-request-in-timingsimplecpu">24.21.2.2.2. gem5 <code>Request</code> in <code>TimingSimpleCPU</code></a></h6>
<div class="paragraph">
<p>In <a href="#gem5-timingsimplecpu">TimingSimpleCPU</a>, the request gets created per memory read:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault
TimingSimpleCPU::initiateMemRead(Addr addr, unsigned size,
                                 Request::Flags flags,
                                 const std::vector&lt;bool&gt;&amp; byte_enable)
{
    ...
    RequestPtr req = std::make_shared&lt;Request&gt;(
        addr, size, flags, dataMasterId(), pc, thread-&gt;contextId());</pre>
</div>
</div>
<div class="paragraph">
<p>and from <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">gem5 functional vs atomic vs timing memory requests</a> and <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">gem5 functional vs atomic vs timing memory requests</a> we remember that <code>initiateMemRead</code> is actually started from the <code>initiateAcc</code> instruction definitions for timing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault LDRWL64_LIT::initiateAcc(ExecContext *xc,
    Trace::InstRecord *traceData) const
{
    ...
    fault = initiateMemRead(xc, traceData, EA, Mem, memAccessFlags);</pre>
</div>
</div>
<div class="paragraph">
<p>From this we see that <code>initiateAcc</code> memory instructions are basically extracting the required information for the request, notably the address <code>EA</code> and flags.</p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-mshr"><a class="anchor" href="#gem5-mshr"></a><a class="link" href="#gem5-mshr">24.21.3. gem5 <code>MSHR</code></a></h4>
<div class="paragraph">
<p>Mentioned at: <a href="http://pages.cs.wisc.edu/~swilson/gem5-docs/gem5MemorySystem.html" class="bare">http://pages.cs.wisc.edu/~swilson/gem5-docs/gem5MemorySystem.html</a></p>
</div>
<div class="paragraph">
<p>Each cache object owns a <code>MSHRQueue</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class BaseCache : public ClockedObject
{
    /** Miss status registers */
    MSHRQueue mshrQueue;</pre>
</div>
</div>
<div class="paragraph">
<p><code>BaseCache</code> is the base class of <code>Cache</code> and <code>NoncoherentCache</code>.</p>
</div>
<div class="paragraph">
<p><code>MSHRQueue</code> is a <code>Queue</code> of <code>MSHR</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class MSHRQueue : public Queue&lt;MSHR&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>and Queue is also a gem5 class under <code>src/mem/cache/queue.hh</code>.</p>
</div>
<div class="paragraph">
<p>The MSHR basically keeps track of all information the cache receives, and helps it take appropriate action. I&#8217;m not sure why it is separate form the cache at all, as it is basically performing essential cache bookkeeping.</p>
</div>
<div class="paragraph">
<p>A clear example of MSHR in action can be seen at: <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a>. In that example what happened was:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPU1 writes to an address and it completes</p>
</li>
<li>
<p>CPU2 sends read</p>
</li>
<li>
<p>CPU1 writes to the address again</p>
</li>
<li>
<p>CPU2 snoops the write, and notes it down in its MSHR</p>
</li>
<li>
<p>CPU2 receives a snoop reply for its read, also from CPU1 which has the data and the line becomes valid</p>
</li>
<li>
<p>CPU2 gets its data. But the MSHR remembers that it had also received a write snoop, so it also immediately invalidates that line</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>From this we understand that MSHR is the part of the cache that synchronizes stuff pending snoops and ensures that things get invalidated.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-commmonitor"><a class="anchor" href="#gem5-commmonitor"></a><a class="link" href="#gem5-commmonitor">24.21.4. gem5 <code>CommMonitor</code></a></h4>
<div class="paragraph">
<p>You can place this <a href="#gem5-python-c-interaction">SimObject</a> in between two <a href="#gem5-port-system">ports</a> to get extra statistics about the packets that are going through.</p>
</div>
<div class="paragraph">
<p>It only works on <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">timing requests</a>, and does not seem to dump any memory values, only add extra <a href="#gem5-m5out-stats-txt-file">statistics</a>.</p>
</div>
<div class="paragraph">
<p>For example, the patch <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/patches/manual/gem5-commmonitor-se.patch">patches/manual/gem5-commmonitor-se.patch</a> hack a <code>CommMonitor</code> between the CPU and the L1 cache on top of gem5 1c3662c9557c85f0d25490dc4fbde3f8ab0cb350:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>patch -d "$(./getvar gem5_source_dir)" -p 1 &lt; patches/manual/gem5-commmonitor-se.patch</pre>
</div>
</div>
<div class="paragraph">
<p>That patch was done largely by copying what <code>fs.py --memcheck</code> does with a <code>MemChecker</code> object.</p>
</div>
<div class="paragraph">
<p>You can then run with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  -- \
  --caches \
  --cpu-type TimingSimpleCPU \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and now we have some new extra histogram statistics such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system.cpu.dcache_mon.readBurstLengthHist::samples            1</pre>
</div>
</div>
<div class="paragraph">
<p>One neat thing about this is that it is agnostic to the memory object type, so you don&#8217;t have to recode those statistics for every new type of object that operates on memory packets.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-simplememory"><a class="anchor" href="#gem5-simplememory"></a><a class="link" href="#gem5-simplememory">24.21.5. gem5 <code>SimpleMemory</code></a></h4>
<div class="paragraph">
<p><code>SimpleMemory</code> is a highly simplified memory system. It can replace a more complex DRAM model if you use it e.g. as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 -- --mem-type SimpleMemory</pre>
</div>
</div>
<div class="paragraph">
<p>and it also gets used in certain system-y memories present in ARM systems by default e.g. Flash memory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.realview.flash0]
type=SimpleMemory</pre>
</div>
</div>
<div class="paragraph">
<p>As of gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 LKMC 059a7ef9d9c378a6d1d327ae97d90b78183680b2 it did not provide any speedup to the Linux kernel boot according to a quick test.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-internals"><a class="anchor" href="#gem5-internals"></a><a class="link" href="#gem5-internals">24.22. gem5 internals</a></h3>
<div class="paragraph">
<p>Internals under other sections:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-memory-system">gem5 memory system</a></p>
</li>
<li>
<p><a href="#gem5-trace-internals">gem5 trace internals</a></p>
</li>
<li>
<p><a href="#gem5-checkpoint-internals">gem5 checkpoint internals</a></p>
</li>
<li>
<p><a href="#gem5-graphic-mode-internals">gem5 graphic mode internals</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gem5-eclipse-configuration"><a class="anchor" href="#gem5-eclipse-configuration"></a><a class="link" href="#gem5-eclipse-configuration">24.22.1. gem5 Eclipse configuration</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/61656709/how-to-setup-eclipse-ide-for-gem5-development" class="bare">https://stackoverflow.com/questions/61656709/how-to-setup-eclipse-ide-for-gem5-development</a></p>
</div>
<div class="paragraph">
<p>In order to develop complex C++ software such as gem5, a good IDE setup is fundamental.</p>
</div>
<div class="paragraph">
<p>The best setup I&#8217;ve reached is with Eclipse. It is not perfect, and there is a learning curve, but is worth it.</p>
</div>
<div class="paragraph">
<p>Notably, it is very hard to get perfect due to: <a href="#why-are-all-c-symlinked-into-the-gem5-build-dir">Why are all C++ symlinked into the gem5 build dir?</a>.</p>
</div>
<div class="paragraph">
<p>I recommend the following settings, tested in Eclipse 2019.09, Ubuntu 18.04:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>fix all missing stdlib headers: <a href="https://stackoverflow.com/questions/10373788/how-to-solve-unresolved-inclusion-iostream-in-a-c-file-in-eclipse-cdt/51099533#51099533" class="bare">https://stackoverflow.com/questions/10373788/how-to-solve-unresolved-inclusion-iostream-in-a-c-file-in-eclipse-cdt/51099533#51099533</a></p>
</li>
<li>
<p>use spaces instead of tabs: Window, Preferences, Code Style, C/C++, Formatter, New, Edit, Tab Policy, Spaces Only</p>
</li>
<li>
<p>either</p>
<div class="ulist">
<ul>
<li>
<p>create the project in the gem5 build directory! Files are moved around there and symlinked, and this gives the best chances of success</p>
</li>
<li>
<p>add to the include search path:</p>
<div class="ulist">
<ul>
<li>
<p>./src/ in the source tree</p>
</li>
<li>
<p>the ISA specific build directory which contains some self-generated stuff, e.g.: out/gem5/default/build/ARM</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>To run and GDB step debug the executable, just copy the <a href="#dry-run">full command line without newlines</a> from your run command (Eclipse does not like newlines for the arguments), e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --emulator gem5 --print-cmd-oneline</pre>
</div>
</div>
<div class="paragraph">
<p>and configure it into Eclipse as usual.</p>
</div>
<div class="paragraph">
<p>One downside of this setup is that if you want to nuke your build directory to get a clean build, then the Eclipse configuration files present in it might get deleted. Maybe it is possible to store configuration files outside of the directory, but we are now mitigating that by making a backup copy of those configuration files before removing the directory, and restoring it when you do <code>./build-gem --clean</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-python-c-interaction"><a class="anchor" href="#gem5-python-c-interaction"></a><a class="link" href="#gem5-python-c-interaction">24.22.2. gem5 Python C++ interaction</a></h4>
<div class="paragraph">
<p>The interaction uses the Python C extension interface <a href="https://docs.python.org/2/extending/extending.html" class="bare">https://docs.python.org/2/extending/extending.html</a> interface through the <a href="#pybind11">pybind11</a> helper library: <a href="https://github.com/pybind/pybind11" class="bare">https://github.com/pybind/pybind11</a></p>
</div>
<div class="paragraph">
<p>The C++ executable both:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>starts running the Python executable</p>
</li>
<li>
<p>provides Python classes written in C++ for that Python code to use</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>An example of this can be found at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://docs.python.org/2/extending/embedding.html#extending-embedded-python" class="bare">https://docs.python.org/2/extending/embedding.html#extending-embedded-python</a></p>
</li>
<li>
<p><a href="https://github.com/pybind/pybind11/tree/v2.2.3/tests/test_embed" class="bare">https://github.com/pybind/pybind11/tree/v2.2.3/tests/test_embed</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>then gem5 magic <code>SimObject</code> class adds some crazy stuff on top of it further, is is a mess. In particular, it auto generates <code>params/</code> headers. TODO: why is this mess needed at all? pybind11 seems to handle constructor arguments just fine:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/pybind/pybind11/blob/v2.2.3/tests/test_class.py#L77" class="bare">https://github.com/pybind/pybind11/blob/v2.2.3/tests/test_class.py#L77</a></p>
</li>
<li>
<p><a href="https://github.com/pybind/pybind11/blob/v2.2.3/tests/test_class.cpp#L41" class="bare">https://github.com/pybind/pybind11/blob/v2.2.3/tests/test_class.cpp#L41</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Let&#8217;s study <code>BadDevice</code> for example:</p>
</div>
<div class="paragraph">
<p><code>src/dev/BadDevice.py</code> defines <code>devicename</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class BadDevice(BasicPioDevice):
    type = 'BadDevice'
    cxx_header = "dev/baddev.hh"
    devicename = Param.String("Name of device to error on")</pre>
</div>
</div>
<div class="paragraph">
<p>The object is created in Python for example from <code>src/dev/alpha/Tsunami.py</code> as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    fb = BadDevice(pio_addr=0x801fc0003d0, devicename='FrameBuffer')</pre>
</div>
</div>
<div class="paragraph">
<p>Since <code>BadDevice</code> has no <code>__init__</code> method, and neither <code>BasicPioDevice</code>, it all just falls through until the <code>SimObject.__init__</code> constructor.</p>
</div>
<div class="paragraph">
<p>This constructor will loop through the inheritance chain and give the Python parameters to the C++ BadDeviceParams class as follows.</p>
</div>
<div class="paragraph">
<p>The auto-generated <code>build/ARM/params/BadDevice.hh</code> file defines BadDeviceParams in C++:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#ifndef __PARAMS__BadDevice__
#define __PARAMS__BadDevice__

class BadDevice;

#include &lt;cstddef&gt;
#include &lt;string&gt;

#include "params/BasicPioDevice.hh"

struct BadDeviceParams
    : public BasicPioDeviceParams
{
    BadDevice * create();
    std::string devicename;
};

#endif // __PARAMS__BadDevice__</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>./python/_m5/param_BadDevice.cc</code> defines the param Python from C++ with pybind11:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>namespace py = pybind11;

static void
module_init(py::module &amp;m_internal)
{
    py::module m = m_internal.def_submodule("param_BadDevice");
    py::class_&lt;BadDeviceParams, BasicPioDeviceParams, std::unique_ptr&lt;BadDeviceParams, py::nodelete&gt;&gt;(m, "BadDeviceParams")
        .def(py::init&lt;&gt;())
        .def("create", &amp;BadDeviceParams::create)
        .def_readwrite("devicename", &amp;BadDeviceParams::devicename)
        ;

    py::class_&lt;BadDevice, BasicPioDevice, std::unique_ptr&lt;BadDevice, py::nodelete&gt;&gt;(m, "BadDevice")
        ;

}

static EmbeddedPyBind embed_obj("BadDevice", module_init, "BasicPioDevice");</pre>
</div>
</div>
<div class="paragraph">
<p><code>src/dev/baddev.hh</code> then uses the parameters on the constructor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class BadDevice : public BasicPioDevice
{
  private:
    std::string devname;

  public:
    typedef BadDeviceParams Params;

  protected:
    const Params *
    params() const
    {
        return dynamic_cast&lt;const Params *&gt;(_params);
    }

  public:
     /**
      * Constructor for the Baddev Class.
      * @param p object parameters
      * @param a base address of the write
      */
    BadDevice(Params *p);</pre>
</div>
</div>
<div class="paragraph">
<p><code>src/dev/baddev.cc</code> then uses the parameter:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BadDevice::BadDevice(Params *p)
    : BasicPioDevice(p, 0x10), devname(p-&gt;devicename)
{
}</pre>
</div>
</div>
<div class="paragraph">
<p>It has been found that this usage of <a href="#pybind11">pybind11</a> across hundreds of <code>SimObject</code> files accounted for 50% of the gem5 build time at one point: <a href="#pybind11-accounts-for-50-of-gem5-build-time">pybind11 accounts for 50% of gem5 build time</a>.</p>
</div>
<div class="paragraph">
<p>To get a feeling of how <code>SimObject</code> objects are run, see: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/61910993/viewing-the-parameters-of-the-branch-predictor-in-gem5/61914449#61914449" class="bare">https://stackoverflow.com/questions/61910993/viewing-the-parameters-of-the-branch-predictor-in-gem5/61914449#61914449</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/62969566/attributes-of-system-object-in-gem5/62970092#62970092" class="bare">https://stackoverflow.com/questions/62969566/attributes-of-system-object-in-gem5/62970092#62970092</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested on gem5 08c79a194d1a3430801c04f37d13216cc9ec1da3.</p>
</div>
</div>
<div class="sect3">
<h4 id="gem5-entry-point"><a class="anchor" href="#gem5-entry-point"></a><a class="link" href="#gem5-entry-point">24.22.3. gem5 entry point</a></h4>
<div class="paragraph">
<p>The main is at: <code>src/sim/main.cc</code>. It calls:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ret = initM5Python();</pre>
</div>
</div>
<div class="paragraph">
<p>src/sim/init.cc:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>230 int
231 initM5Python()
232 {
233     EmbeddedPyBind::initAll();
234     return EmbeddedPython::initAll();
235 }</pre>
</div>
</div>
<div class="paragraph">
<p><code>initAll</code> basically just initializes the <code>_m5</code> Python object, which is used across multiple <code>.py</code>.</p>
</div>
<div class="paragraph">
<p>Back on <code>main</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ret = m5Main(argc, argv);</pre>
</div>
</div>
<div class="paragraph">
<p>which goes to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>result = PyRun_String(*command, Py_file_input, dict, dict);</pre>
</div>
</div>
<div class="paragraph">
<p>with commands looping over:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>import m5
m5.main()</pre>
</div>
</div>
<div class="paragraph">
<p>which leads into:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/python/m5/main.py#main</pre>
</div>
</div>
<div class="paragraph">
<p>which finally calls your config file like <code>fs.py</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>filename = sys.argv[0]
filedata = file(filename, 'r').read()
filecode = compile(filedata, filename, 'exec')
[...]
exec filecode in scope</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: the file path name appears to be passed as a command line argument to the Python script, but I didn&#8217;t have the patience to fully understand the details.</p>
</div>
<div class="paragraph">
<p>The Python config files then set the entire system up in Python, and finally call <code>m5.simulate()</code> to run the actual simulation. This function has a C++ native implementation at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/sim/simulate.cc</pre>
</div>
</div>
<div class="paragraph">
<p>and that is where the main event loop, <code>doSimLoop</code>, gets called and starts kicking off the <a href="#gem5-event-queue">gem5 event queue</a>.</p>
</div>
<div class="paragraph">
<p>Tested at gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
</div>
<div class="sect4">
<h5 id="gem5-m5-objects-module"><a class="anchor" href="#gem5-m5-objects-module"></a><a class="link" href="#gem5-m5-objects-module">24.22.3.1. gem5 <code>m5.objects</code> module</a></h5>
<div class="paragraph">
<p>All <code>SimObjects</code> seem to be automatically added to the <code>m5.objects</code> namespace, and this is done in a very convoluted way, let&#8217;s try to understand a bit:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/python/m5/objects/__init__.py</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>modules = __loader__.modules

for module in modules.keys():
    if module.startswith('m5.objects.'):
        exec("from %s import *" % module)</pre>
</div>
</div>
<div class="paragraph">
<p>And from <a href="#debug-gem5-python-scripts">IPDB</a> we see that this appears to loop over every object string of type <code>m5.objects.modulename</code>.</p>
</div>
<div class="paragraph">
<p>This <code>__init__</code> gets called from <code>src/python/importer.py</code> at the <code>exec</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class CodeImporter(object):
    def load_module(self, fullname):
            override = os.environ.get('M5_OVERRIDE_PY_SOURCE', 'false').lower()
            if override in ('true', 'yes') and  os.path.exists(abspath):
                src = open(abspath, 'r').read()
                code = compile(src, abspath, 'exec')

            if os.path.basename(srcfile) == '__init__.py':
                mod.__path__ = fullname.split('.')
                mod.__package__ = fullname
            else:
                mod.__package__ = fullname.rpartition('.')[0]
            mod.__file__ = srcfile

            exec(code, mod.__dict__)

import sys
importer = CodeImporter()
add_module = importer.add_module
sys.meta_path.append(importer)</pre>
</div>
</div>
<div class="paragraph">
<p>Here as a bonus here we also see how <a href="#m5-override-py-source"><code>M5_OVERRIDE_PY_SOURCE</code></a> works.</p>
</div>
<div class="paragraph">
<p>In <code>src/SConscript</code> we see that <code>SimObject</code> is just a <code>PySource</code> with module equals to <code>m5.objects</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class SimObject(PySource):
    def __init__(self, source, tags=None, add_tags=None):
        '''Specify the source file and any tags (automatically in
        the m5.objects package)'''
        super(SimObject, self).__init__('m5.objects', source, tags, add_tags)</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>add_module</code> method seems to be doing the magic and is called from <code>src/sim/init.cc</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>bool
EmbeddedPython::addModule() const
{
    PyObject *code = getCode();
    PyObject *result = PyObject_CallMethod(importerModule, PyCC("add_module"),</pre>
</div>
</div>
<div class="paragraph">
<p>which is called from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>int
EmbeddedPython::initAll()
{
    // Load the importer module
    PyObject *code = importer-&gt;getCode();
    importerModule = PyImport_ExecCodeModule(PyCC("importer"), code);
    if (!importerModule) {
        PyErr_Print();
        return 1;
    }

    // Load the rest of the embedded python files into the embedded
    // python importer
    list&lt;EmbeddedPython *&gt;::iterator i = getList().begin();
    list&lt;EmbeddedPython *&gt;::iterator end = getList().end();
    for (; i != end; ++i)
        if (!(*i)-&gt;addModule())</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>getList</code> comes from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EmbeddedPython::EmbeddedPython(const char *filename, const char *abspath,
    const char *modpath, const unsigned char *code, int zlen, int len)
    : filename(filename), abspath(abspath), modpath(modpath), code(code),
      zlen(zlen), len(len)
{
    // if we've added the importer keep track of it because we need it
    // to bootstrap.
    if (string(modpath) == string("importer"))
        importer = this;
    else
        getList().push_back(this);
}

list&lt;EmbeddedPython *&gt; &amp;
EmbeddedPython::getList()
{
    static list&lt;EmbeddedPython *&gt; the_list;
    return the_list;
}</pre>
</div>
</div>
<div class="paragraph">
<p>and the constructor in turn gets called from per <code>SimObject</code> autogenerated files such as e.g. <code>dev/storage/Ide.py.cc</code> for <code>src/dev/storage/Ide.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EmbeddedPython embedded_m5_objects_Ide(
    "m5/objects/Ide.py",
    "/home/ciro/bak/git/linux-kernel-module-cheat/data/gem5/master4/src/dev/storage/Ide.py",
    "m5.objects.Ide",
    data_m5_objects_Ide,
    947,
    2099);

} // anonymous namespace</pre>
</div>
</div>
<div class="paragraph">
<p>which get autogenerated at <code>src/SConscript</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def embedPyFile(target, source, env):

for source in PySource.all:
    base_py_env.Command(source.cpp, [ py_marshal, source.tnode ],
                        MakeAction(embedPyFile, Transform("EMBED PY")))</pre>
</div>
</div>
<div class="paragraph">
<p>where the <code>PySource.all</code> thing as you might expect is a static list of all <code>PySource</code> source files as they get updated in the constructor.</p>
</div>
<div class="paragraph">
<p>Tested in gem5 d9cb548d83fa81858599807f54b52e5be35a6b03.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-event-queue"><a class="anchor" href="#gem5-event-queue"></a><a class="link" href="#gem5-event-queue">24.22.4. gem5 event queue</a></h4>
<div class="paragraph">
<p>gem5 is an event based simulator, and as such the event queue is of of the crucial elements in the system.</p>
</div>
<div class="paragraph">
<p>Every single action that takes time (e.g. notably <a href="#timingsimplecpu-analysis-ldr-stall">reading from memory</a>) models that time delay by scheduling an event in the future.</p>
</div>
<div class="paragraph">
<p>The gem5 event queue stores one callback event for each future point in time.</p>
</div>
<div class="paragraph">
<p>The event queue is implemented in the class <code>EventQueue</code> in the file <code>src/sim/eventq.hh</code>.</p>
</div>
<div class="paragraph">
<p>Not all times need to have an associated event: if a given time has no events, gem5 just skips it and jumps to the next event: the queue is basically a linked list of events.</p>
</div>
<div class="paragraph">
<p>Important examples of events include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPU ticks</p>
</li>
<li>
<p>peripherals and memory</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>At <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis</a> we see for example that at the beginning of an <a href="#gem5-atomicsimplecpu">AtomicCPU</a> simulation, gem5 sets up exactly two events:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the first CPU cycle</p>
</li>
<li>
<p>one exit event at the end of time which triggers <a href="#gem5-simulate-limit-reached">gem5 simulate() limit reached</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, at the end of the callback of one tick event, another tick is scheduled.</p>
</div>
<div class="paragraph">
<p>And so the simulation progresses tick by tick, until an exit event happens.</p>
</div>
<div class="paragraph">
<p>The <code>EventQueue</code> class has one awesome <code>dump()</code> function that prints a human friendly representation of the queue, and can be easily called from GDB. TODO example.</p>
</div>
<div class="paragraph">
<p>We can also observe what is going on in the event queue with the <code>Event</code> <a href="#gem5-tracing">debug flag</a>.</p>
</div>
<div class="paragraph">
<p>Event execution is done at <code>EventQueue::serviceOne()</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Event *exit_event = eventq-&gt;serviceOne();</pre>
</div>
</div>
<div class="paragraph">
<p>This calls the <code>Event::process</code> method of the event.</p>
</div>
<div class="paragraph">
<p>Another important technique is to use <a href="#debug-the-emulator">GDB</a> and break at interesting points such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>b Trace::OstreamLogger::logMessage
b EventManager::schedule
b EventFunctionWrapper::process</pre>
</div>
</div>
<div class="paragraph">
<p>although stepping into <code>EventFunctionWrapper::process</code> which does <code>std::function</code> is a bit of a pain: <a href="https://stackoverflow.com/questions/59429401/how-to-step-into-stdfunction-user-code-from-c-functional-with-gdb" class="bare">https://stackoverflow.com/questions/59429401/how-to-step-into-stdfunction-user-code-from-c-functional-with-gdb</a></p>
</div>
<div class="paragraph">
<p>Another potentially useful technique is to use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--trace Event,ExecAll,FmtFlag,FmtStackTrace --trace-stdout</pre>
</div>
</div>
<div class="paragraph">
<p>which automates the logging of <code>Trace::OstreamLogger::logMessage()</code> backtraces.</p>
</div>
<div class="paragraph">
<p>But alas, it misses which function callback is being scheduled, which is the awesome thing we actually want:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/37545327/get-the-name-of-a-stdfunction" class="bare">https://stackoverflow.com/questions/37545327/get-the-name-of-a-stdfunction</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/40706805/how-to-convert-a-function-pointer-to-function-name/40706869" class="bare">https://stackoverflow.com/questions/40706805/how-to-convert-a-function-pointer-to-function-name/40706869</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then, once we had that, the most perfect thing ever would be to make the full event graph containing which events schedule which events!</p>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis"><a class="anchor" href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis"></a><a class="link" href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis">24.22.4.1. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis</a></h5>
<div class="paragraph">
<p>Let&#8217;s now analyze every single event on a minimal <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a> in the <a href="#gem5-cpu-types">simplest CPU that we have</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace Event,ExecAll,FmtFlag \
  --trace-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 scheduled @ 0
**** REAL SIMULATION ****
      0: Event: Event_70: generic 70 scheduled @ 0
info: Entering event queue @ 0.  Starting simulation...
      0: Event: Event_70: generic 70 rescheduled @ 18446744073709551615
      0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 0
      0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
      0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 500
    500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 500
    500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
    500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 1000
   1000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 1000
   1000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)
   1000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 1500
   1500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 1500
   1500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x8, #64, #0       : IntAlu :  D=0x0000000000000040  flags=(IsInteger)
   1500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 2000
   2000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 2000
   2000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
hello
   2000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 2500
   2500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 2500
   2500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   2500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 3000
   3000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 3000
   3000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
   3000: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 3500
   3500: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 3500
   3500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
   3500: Event: Event_71: generic 71 scheduled @ 3500
   3500: Event: Event_71: generic 71 executed @ 3500</pre>
</div>
</div>
<div class="paragraph">
<p>On the event trace, we can first see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 scheduled @ 0</pre>
</div>
</div>
<div class="paragraph">
<p>This schedules a tick event for time <code>0</code>, and leads to the first clock tick.</p>
</div>
<div class="paragraph">
<p>Then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0: Event: Event_70: generic 70 scheduled @ 0
0: Event: Event_70: generic 70 rescheduled @ 18446744073709551615</pre>
</div>
</div>
<div class="paragraph">
<p>schedules the end of time event for time <code>0</code>, which is later rescheduled to the actual end of time.</p>
</div>
<div class="paragraph">
<p>At:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 executed @ 0
0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 rescheduled @ 500</pre>
</div>
</div>
<div class="paragraph">
<p>the tick event happens, the instruction runs, and then the instruction is rescheduled in <code>500</code> time units. This is done at the end of <code>AtomicSimpleCPU::tick()</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if (_status != Idle)
    reschedule(tickEvent, curTick() + latency, true);</pre>
</div>
</div>
<div class="paragraph">
<p>At:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>3500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
3500: Event: Event_71: generic 71 scheduled @ 3500
3500: Event: Event_71: generic 71 executed @ 3500</pre>
</div>
</div>
<div class="paragraph">
<p>the exit system call is called, and then it schedules an exit evit, which gets executed and the simulation ends.</p>
</div>
<div class="paragraph">
<p>We guess then that <code>Event_71</code> comes from the SE implementation of the exit syscall, so let&#8217;s just confirm, the trace contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>exitSimLoop() at sim_events.cc:97 0x5555594746e0
exitImpl() at syscall_emul.cc:215 0x55555948c046
exitFunc() at syscall_emul.cc:225 0x55555948c147
SyscallDesc::doSyscall() at syscall_desc.cc:72 0x5555594949b6
Process::syscall() at process.cc:401 0x555559484717
SimpleThread::syscall() at 0x555559558059
ArmISA::SupervisorCall::invoke() at faults.cc:856 0x5555572950d7
BaseSimpleCPU::advancePC() at base.cc:681 0x555559083133
AtomicSimpleCPU::tick() at atomic.cc:757 0x55555907834c</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>exitSimLoop()</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>new GlobalSimLoopExitEvent(when + simQuantum, message, exit_code, repeat);</pre>
</div>
</div>
<div class="paragraph">
<p>Tested in gem5 12c917de54145d2d50260035ba7fa614e25317a3.</p>
</div>
<div class="sect5">
<h6 id="atomicsimplecpu-initial-events"><a class="anchor" href="#atomicsimplecpu-initial-events"></a><a class="link" href="#atomicsimplecpu-initial-events">24.22.4.1.1. AtomicSimpleCPU initial events</a></h6>
<div class="paragraph">
<p>Let&#8217;s have a closer look at the initial magically scheduled events of the simulation.</p>
</div>
<div class="paragraph">
<p>Most events come from other events, but at least one initial event must be scheduled somehow from elsewhere to kick things off.</p>
</div>
<div class="paragraph">
<p>The initial tick event:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0: Event: AtomicSimpleCPU tick.wrapped_function_event: EventFunctionWrapped 39 scheduled @ 0</pre>
</div>
</div>
<div class="paragraph">
<p>we&#8217;ll study by breaking at at the point that prints messages: <code>b Trace::OstreamLogger::logMessage()</code> to see where events are being scheduled from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Trace::OstreamLogger::logMessage() at trace.cc:149 0x5555593b3b1e
void Trace::Logger::dprintf_flag&lt;char const*, char const*, unsigned long&gt;() at 0x55555949e603
void Trace::Logger::dprintf&lt;char const*, char const*, unsigned long&gt;() at 0x55555949de58
Event::trace() at eventq.cc:395 0x55555946d109
EventQueue::schedule() at eventq_impl.hh:65 0x555557195441
EventManager::schedule() at eventq.hh:746 0x555557194aa2
AtomicSimpleCPU::activateContext() at atomic.cc:239 0x555559075531
SimpleThread::activate() at simple_thread.cc:177 0x555559545a63
Process::initState() at process.cc:283 0x555559484011
ArmProcess64::initState() at process.cc:126 0x55555730827a
ArmLinuxProcess64::initState() at process.cc:1,777 0x5555572d5e5e</pre>
</div>
</div>
<div class="paragraph">
<p>The interesting call is at <code>AtomicSimpleCPU::activateContext</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>schedule(tickEvent, clockEdge(Cycles(0)));</pre>
</div>
</div>
<div class="paragraph">
<p>which calls <code>EventManager::schedule</code>.</p>
</div>
<div class="paragraph">
<p><code>AtomicSimpleCPU</code> is an <code>EventManager</code> because <a href="#gem5-python-c-interaction"><code>SimObject</code></a> inherits from it.</p>
</div>
<div class="paragraph">
<p><code>tickEvent</code> is an <code>EventFunctionWrapper</code> which contains a <code>std::function&lt;void(void)&gt; callback;</code>, and is initialized in the constructor as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>tickEvent([this]{ tick(); }, "AtomicSimpleCPU tick",
        false, Event::CPU_Tick_Pri),</pre>
</div>
</div>
<div class="paragraph">
<p>The call stack above <code>ArmLinuxProcess64::initState</code> is <a href="#pybind11">pybind11</a> fuzziness, but if we grep a bit we find the Python call point:</p>
</div>
<div class="paragraph">
<p>src/python/m5/simulate.py</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def instantiate(ckpt_dir=None):

    ...

    # Create the C++ sim objects and connect ports
    for obj in root.descendants(): obj.createCCObject()
    for obj in root.descendants(): obj.connectPorts()

    # Do a second pass to finish initializing the sim objects
    for obj in root.descendants(): obj.init()

    ...

    # Restore checkpoint (if any)
    if ckpt_dir:
        ...
    else:
        for obj in root.descendants(): obj.initState()</pre>
</div>
</div>
<div class="paragraph">
<p>and this gets called from the toplevel Python scripts e.g. se.py <code>configs/common/Simulation.py</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>m5.instantiate(checkpoint_dir)</pre>
</div>
</div>
<div class="paragraph">
<p>As we can see, <code>initState</code> is just one stage of generic <code>SimObject</code> initialization. <code>root.descendants()</code> goes over the entire <code>SimObject</code> tree calling <code>initState()</code>.</p>
</div>
<div class="paragraph">
<p>Finally, we see that <code>initState</code> is part of the <code>SimObject</code> C++ API:</p>
</div>
<div class="paragraph">
<p>src/sim/sim_object.hh</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class SimObject : public EventManager, public Serializable, public Drainable,
                  public Stats::Group
{

    ...

    /**
     * initState() is called on each SimObject when *not* restoring
     * from a checkpoint.  This provides a hook for state
     * initializations that are only required for a "cold start".
     */
    virtual void initState();</pre>
</div>
</div>
<div class="paragraph">
<p>Finally, we see that <code>initState</code> is exposed to the Python API at:</p>
</div>
<div class="paragraph">
<p>build/ARM/python/_m5/param_SimObject.cc</p>
</div>
<div class="literalblock">
<div class="content">
<pre>module_init(py::module &amp;m_internal)
{
    py::module m = m_internal.def_submodule("param_SimObject");
    py::class_&lt;SimObjectParams, std::unique_ptr&lt;SimObjectParams, py::nodelete&gt;&gt;(m, "SimObjectParams")
        .def_readwrite("name", &amp;SimObjectParams::name)
        .def_readwrite("eventq_index", &amp;SimObjectParams::eventq_index)
        ;

    py::class_&lt;SimObject, Drainable, Serializable, Stats::Group, std::unique_ptr&lt;SimObject, py::nodelete&gt;&gt;(m, "SimObject")
        .def("init", &amp;SimObject::init)
        .def("initState", &amp;SimObject::initState)
        .def("memInvalidate", &amp;SimObject::memInvalidate)
        .def("memWriteback", &amp;SimObject::memWriteback)
        .def("regProbePoints", &amp;SimObject::regProbePoints)
        .def("regProbeListeners", &amp;SimObject::regProbeListeners)
        .def("startup", &amp;SimObject::startup)
        .def("loadState", &amp;SimObject::loadState, py::arg("cp"))
        .def("getPort", &amp;SimObject::getPort, pybind11::return_value_policy::reference, py::arg("if_name"), py::arg("idx"))
        ;

}</pre>
</div>
</div>
<div class="paragraph">
<p>which is more magical than the other param classes since <code>py::class_&lt;SimObject</code> has non-trivial methods, those are auto-generated by the <code>cxx_exports</code> code generation mechanism:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class SimObject(object):

    ...

    cxx_exports = [
        PyBindMethod("init"),
        PyBindMethod("initState"),
        PyBindMethod("memInvalidate"),
        PyBindMethod("memWriteback"),
        PyBindMethod("regProbePoints"),
        PyBindMethod("regProbeListeners"),
        PyBindMethod("startup"),
    ]</pre>
</div>
</div>
<div class="paragraph">
<p>And the second magically scheduled event is the exit event:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0: Event: Event_70: generic 70 scheduled @ 0
0: Event: Event_70: generic 70 rescheduled @ 18446744073709551615</pre>
</div>
</div>
<div class="paragraph">
<p>which is scheduled with backtrace:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Trace::OstreamLogger::logMessage() at trace.cc:149 0x5555593b3b1e
void Trace::Logger::dprintf_flag&lt;char const*, char const*, unsigned long&gt;() at 0x55555949e603
void Trace::Logger::dprintf&lt;char const*, char const*, unsigned long&gt;() at 0x55555949de58
Event::trace() at eventq.cc:395 0x55555946d109
EventQueue::schedule() at eventq_impl.hh:65 0x555557195441
BaseGlobalEvent::schedule() at global_event.cc:78 0x55555946d6f1
GlobalEvent::GlobalEvent() at 0x55555949d177
GlobalSimLoopExitEvent::GlobalSimLoopExitEvent() at sim_events.cc:61 0x555559474470
simulate() at simulate.cc:104 0x555559476d6f</pre>
</div>
</div>
<div class="paragraph">
<p>which comes at object creation inside <code>simulate()</code> through the <code>GlobalEvent()</code> constructor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>simulate_limit_event =
    new GlobalSimLoopExitEvent(mainEventQueue[0]-&gt;getCurTick(),
                                "simulate() limit reached", 0);</pre>
</div>
</div>
<div class="paragraph">
<p>This event indicates that the simulation should finish by overriding <code>bool isExitEvent()</code> which gets checked in the main simulation at <code>EventQueue::serviceOne</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if (event-&gt;isExitEvent()) {
    assert(!event-&gt;flags.isSet(Event::Managed) ||
            !event-&gt;flags.isSet(Event::IsMainQueue)); // would be silly
    return event;</pre>
</div>
</div>
<div class="paragraph">
<p>Tested in gem5 12c917de54145d2d50260035ba7fa614e25317a3.</p>
</div>
</div>
<div class="sect5">
<h6 id="atomicsimplecpu-tick-reschedule-timing"><a class="anchor" href="#atomicsimplecpu-tick-reschedule-timing"></a><a class="link" href="#atomicsimplecpu-tick-reschedule-timing">24.22.4.1.2. AtomicSimpleCPU tick reschedule timing</a></h6>
<div class="paragraph">
<p>Inside <code>AtomicSimpleCPU::tick()</code> we saw previously that the reschedule happens at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    if (latency &lt; clockPeriod())
        latency = clockPeriod();

    if (_status != Idle)
        reschedule(tickEvent, curTick() + latency, true);</pre>
</div>
</div>
<div class="paragraph">
<p>so it is interesting to learn where that <code>latency</code> comes from.</p>
</div>
<div class="paragraph">
<p>From our logs, we see that all events happened with a <code>500</code> time unit interval between them, so that must be the value for all instructions of our simple example.</p>
</div>
<div class="paragraph">
<p>By GDBing it a bit, we see that none of our instructions incremented <code>latency</code>, and so it got set to <code>clockPeriod()</code>, which comes from <code>ClockDomain::clockPeriod()</code> which then likely comes from:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    parser.add_option("--cpu-clock", action="store", type="string",
                      default='2GHz',</pre>
</div>
</div>
<div class="paragraph">
<p>because the time unit is picoseconds. This then shows on the <a href="#gem5-config-ini"><code>config.ini</code></a> as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu_clk_domain]
type=SrcClockDomain
clock=500</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="atomicsimplecpu-memory-access"><a class="anchor" href="#atomicsimplecpu-memory-access"></a><a class="link" href="#atomicsimplecpu-memory-access">24.22.4.1.3. AtomicSimpleCPU memory access</a></h6>
<div class="paragraph">
<p>It will be interesting to see how <code>AtomicSimpleCPU</code> makes memory access on GDB and to compare that with <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis"><code>TimingSimpleCPU</code></a>.</p>
</div>
<div class="paragraph">
<p>We assume that the memory access still goes through the <a href="#gem5-crossbar-interconnect"><code>CoherentXBar</code></a>, but instead of generating an event to model delayed response, it must be doing the access directly.</p>
</div>
<div class="paragraph">
<p>Inside <code>AtomicSimpleCPU::tick</code>, we track <code>ifetch_req</code> and see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fault = thread-&gt;itb-&gt;translateAtomic(ifetch_req, thread-&gt;getTC(),
                                        BaseTLB::Execute);</pre>
</div>
</div>
<div class="paragraph">
<p>and later on after translation the memory is obtained at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>icache_latency = sendPacket(icachePort, &amp;ifetch_pkt);</pre>
</div>
</div>
<div class="paragraph">
<p>which <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">sends the packet atomically</a> through the port:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>AtomicSimpleCPU::sendPacket(MasterPort &amp;port, const PacketPtr &amp;pkt) {
    return port.sendAtomic(pkt);
}</pre>
</div>
</div>
<div class="paragraph">
<p>We can compare that with what happen sin <code>TimingSimpleCPU</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>thread-&gt;itb-&gt;translateTiming(ifetch_req, thread-&gt;getTC(),
        &amp;fetchTranslation, BaseTLB::Execute);</pre>
</div>
</div>
<div class="paragraph">
<p>and so there it is: the <code>ITB</code> classes are the same, but there are a separate <code>Atomic</code> and <code>Timing</code> methods!</p>
</div>
<div class="paragraph">
<p>The timing request is shown further at: <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">sends the packet atomically</a>.</p>
</div>
<div class="paragraph">
<p>Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-se-py-page-translation"><a class="anchor" href="#gem5-se-py-page-translation"></a><a class="link" href="#gem5-se-py-page-translation">24.22.4.1.4. gem5 se.py page translation</a></h6>
<div class="paragraph">
<p>Happens on <code>EmulationPageTable</code>, and seems to happen atomically without making any extra memory requests.</p>
</div>
<div class="paragraph">
<p>TODO confirm from code, notably by seeing where the translation table is set.</p>
</div>
<div class="paragraph">
<p>But we can confirm with logging with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--trace DRAM,ExecAll,FmtFlag</pre>
</div>
</div>
<div class="paragraph">
<p>which gives</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x78
      0: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
    500: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x7c
    500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
   1000: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x80
   1000: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0xa0
   1000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)
   1500: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x84
   1500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x8, #64, #0       : IntAlu :  D=0x0000000000000040  flags=(IsInteger)
   2000: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x88
   2000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
hello
   2500: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x8c
   2500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
   3000: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x90
   3000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
   3500: DRAM: system.mem_ctrls: recvAtomic: ReadReq 0x94
   3500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
Exiting @ tick 3500 because exiting with last active thread context
   3500: DRAM: system.mem_ctrls_0: Computing stats due to a dump callback
   3500: DRAM: system.mem_ctrls_1: Computing stats due to a dump callback</pre>
</div>
</div>
<div class="paragraph">
<p>So we see that before every instruction execution there was a DRAM event! Also, each read happens 4 bytes after the previous one, which is consistent with instruction fetches.</p>
</div>
<div class="paragraph">
<p>The DRAM addresses are very close to zero e.g. <code>0x78</code> for the first instruction, and therefore we guess that they are physical since the ELF entry point is much higher:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch aarch64 readelf -- -h "$(./getvar --arch aarch64 userland_build_dir)/arch/aarch64/freestanding/linux/hello.out</pre>
</div>
</div>
<div class="paragraph">
<p>at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  Entry point address:               0x400078</pre>
</div>
</div>
<div class="paragraph">
<p>For LDR, we see that there was an extra DRAM read as well after the fetch read, as expected.</p>
</div>
<div class="paragraph">
<p>Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis"><a class="anchor" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis"></a><a class="link" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">24.22.4.2. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a></h5>
<div class="paragraph">
<p>Now, let&#8217;s move on to <code>TimingSimpleCPU</code>, which is just like <code>AtomicSimpleCPU</code> internally, but now the memory requests don&#8217;t actually finish immediately: <a href="#gem5-cpu-types">gem5 CPU types</a>!</p>
</div>
<div class="paragraph">
<p>This means that simulation will be much more accurate, and the DRAM memory will be modelled.</p>
</div>
<div class="paragraph">
<p>TODO: analyze better what each of the memory event mean. For now, we have just collected a bunch of data there, but needs interpreting. The CPU specifics in this section are already insightful however.</p>
</div>
<div class="paragraph">
<p><a href="#gem5-timingsimplecpu">TimingSimpleCPU</a> should be the second simplest CPU to analyze, so let&#8217;s give it a try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace Event,ExecAll,FmtFlag \
  --trace-stdout \
  -- \
  --cpu-type TimingSimpleCPU \
;</pre>
</div>
</div>
<div class="paragraph">
<p>As of LKMC 78ce2dabe18ef1d87dc435e5bc9369ce82e8d6d2 gem5 12c917de54145d2d50260035ba7fa614e25317a3 the log is now much more complex.</p>
</div>
<div class="paragraph">
<p>Here is an abridged version with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the beginning up to the second instruction</p>
</li>
<li>
<p>end ending</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>because all that happens in between is exactly the same as the first two instructions and therefore boring.</p>
</div>
<div class="paragraph">
<p>We have also manually added:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>double newlines before each event execution</p>
</li>
<li>
<p>line IDs to be able to refer to specific events more easily (<code>#0</code>, <code>#1</code>, etc.)</p>
</li>
</ul>
</div>
<div class="literalblock">
<div class="content">
<pre>#0       0: Event: system.cpu.wrapped_function_event: EventFunctionWrapped 43 scheduled @ 0
**** REAL SIMULATION ****
#1       0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 14 scheduled @ 7786250
#2       0: Event: system.mem_ctrls_1.wrapped_function_event: EventFunctionWrapped 20 scheduled @ 7786250
#3       0: Event: Event_74: generic 74 scheduled @ 0
info: Entering event queue @ 0.  Starting simulation...
#4       0: Event: Event_74: generic 74 rescheduled @ 18446744073709551615

#5       0: Event: system.cpu.wrapped_function_event: EventFunctionWrapped 43 executed @ 0
#6       0: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 0
#7       0: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 scheduled @ 1000

#8       0: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 0
#9       0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 12 scheduled @ 0
#10      0: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 scheduled @ 46250
#11      0: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 5000

#12      0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 12 executed @ 0
#13      0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 15 scheduled @ 0

#14      0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 15 executed @ 0

#15   1000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 executed @ 1000

#16   5000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 5000

#17  46250: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 executed @ 46250
#18  46250: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 scheduled @ 74250

#19  74250: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 executed @ 74250
#20  74250: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 64 scheduled @ 77000
#21  74250: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 65 scheduled @ 77000

#22  77000: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 65 executed @ 77000

#23  77000: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 64 executed @ 77000
#24  77000: Event: Event_40: Timing CPU icache tick 40 scheduled @ 77000

#25  77000: Event: Event_40: Timing CPU icache tick 40 executed @ 77000
     77000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
#26  77000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 77000
#27  77000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 scheduled @ 78000

#28  77000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 77000
#29  77000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 scheduled @ 95750
#30  77000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 77000

#31  77000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 77000

#32  78000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 executed @ 78000

#33  95750: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 executed @ 95750
#34  95750: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 scheduled @ 123750

#35 123750: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 executed @ 123750
#36 123750: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 64 scheduled @ 126000
#37 123750: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 65 scheduled @ 126000

#38 126000: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 65 executed @ 126000

#39 126000: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 64 executed @ 126000
#40 126000: Event: Event_40: Timing CPU icache tick 40 scheduled @ 126000

#41 126000: Event: Event_40: Timing CPU icache tick 40 executed @ 126000
    126000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
#42 126000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 126000
#43 126000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 scheduled @ 127000

    [...]

    469000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
    469000: Event: Event_75: generic 75 scheduled @ 469000
    469000: Event: Event_75: generic 75 executed @ 469000</pre>
</div>
</div>
<div class="paragraph">
<p>Looking into the generated <a href="#gem5-config-ini"><code>config.dot.svg</code></a> can give a better intuition on the shape of the memory system: <a href="#config-dot-svg-timingsimplecpu">Figure 3, &#8220;<code>config.dot.svg</code> for a TimingSimpleCPU without caches.&#8221;</a>, so it is good to keep that in mind.</p>
</div>
<div id="config-dot-svg-timingsimplecpu" class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/media/master/gem5_config_TimingSimpleCPU_12c917de54145d2d50260035ba7fa614e25317a3.svg?sanitize=true" alt="gem5 config TimingSimpleCPU 12c917de54145d2d50260035ba7fa614e25317a3" height="600">
</div>
<div class="title">Figure 3. <code>config.dot.svg</code> for a TimingSimpleCPU without caches.</div>
</div>
<div class="paragraph">
<p>It is also helpful to see this as a tree of events where one execute event schedules other events:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    | | | | |
    0 1 2 3 4   0 TimingSimpleCPU::fetch
    5
    |
    +---+
    |   |
    6   7       6 DRAMCtrl::processNextReqEvent (0)
    8   15      7 BaseXBar::Layer::releaseLayer
    |
+---+---+
|   |   |
9   10  11      9 DRAMCtrl::Rank::processActivateEvent
12  17  16     10 DRAMCtrl::processRespondEvent (46.25)
|   |          11 DRAMCtrl::processNextReqEvent (5)
|   |
13  18         13 DRAMCtrl::Rank::processPowerEvent
14  19         18 PacketQueue::processSendEvent (28)
    |
    +---+
    |   |
    20  21     20 PacketQueue::processSendEvent  (2.75)
    23  22     21 BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer
    |
    24         24 TimingSimpleCPU::IcachePort::ITickEvent::process (0)
    25
    |
    +---+
    |   |
    26  27     26 DRAMCtrl::processNextReqEvent
    28  32     27 BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer
    |
    +---+
    |   |
    29  30     29 DRAMCtrl::processRespondEvent
    33  31     30 DRAMCtrl::processNextReqEvent
    |
    34         34 PacketQueue::processSendEvent
    35
    |
    +---+
    |   |
    36  37     36 PacketQueue::processSendEvent
    39  38     37 BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer
    |
    40         40 TimingSimpleCPU::IcachePort::ITickEvent::process
    41
    |
    +---+
    |   |
    42  43     42 DRAMCtrl::processNextReqEvent
               43 BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer</pre>
</div>
</div>
<div class="paragraph">
<p>Note that every schedule is followed by an execution, so we put them together, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    |   |
    6   7    6 DRAMCtrl::processNextReqEvent (0)
    8   15   7 BaseXBar::Layer::releaseLayer (0)
    |</pre>
</div>
</div>
<div class="paragraph">
<p>means:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>6</code>: schedule <code>DRAMCtrl::processNextReqEvent</code> to run in <code>0</code> ns after the execution that scheduled it</p>
</li>
<li>
<p><code>8</code>: execute <code>DRAMCtrl::processNextReqEvent</code></p>
</li>
<li>
<p><code>7</code>: schedule <code>BaseXBar::Layer::releaseLayer</code> to run in <code>0</code> ns after the execution that scheduled it</p>
</li>
<li>
<p><code>15</code>: execute <code>BaseXBar::Layer::releaseLayer</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>With this, we can focus on going up the event tree from an event of interest until we see what originally caused it!</p>
</div>
<div class="paragraph">
<p>Notably, the above tree contains the execution of the first two instructions.</p>
</div>
<div class="paragraph">
<p>Observe how the events leading up to the second instruction are basically a copy of those of the first one, this is the basic <code>TimingSimpleCPU</code> event loop in action.</p>
</div>
<div class="paragraph">
<p>One line summary of events:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>#5: adds the request to the DRAM queue, and schedules a <code>DRAMCtrl::processNextReqEvent</code> which later sees that request immediately</p>
</li>
<li>
<p>#8: picks up the only request from the DRAM read queue (<code>readQueue</code>) and services that.</p>
<div class="paragraph">
<p>If there were multiple requests, priority arbitration under <code>DRAMCtrl::chooseNext</code> could chose a different one than the first based on packet priorities</p>
</div>
<div class="paragraph">
<p>This puts the request on the response queue <code>respQueue</code> and schedules another <code>DRAMCtrl::processNextReqEvent</code> but the request queue is empty, and that does nos schedule further events</p>
</div>
</li>
<li>
<p>#17: picks up the only request from the DRAM response queue and services that by placing it in yet another queue, and scheduling the <code>PacketQueue::processSendEvent</code> which will later pick up that packet</p>
</li>
<li>
<p>#19: picks up the request from the previous queue, and forwards it to another queue, and schedules yet another <code>PacketQueue::processSendEvent</code></p>
<div class="paragraph">
<p>The current one is the DRAM passing the message to the XBar, and the next <code>processSendEvent</code> is the XBar finally sending it back to the CPU</p>
</div>
</li>
<li>
<p>#23: the XBar port is actually sending the reply back.</p>
<div class="paragraph">
<p>If knows to which CPU core to send the request to because ports keep a map of request to source:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>const auto route_lookup = routeTo.find(pkt-&gt;req);</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-0"><a class="anchor" href="#timingsimplecpu-analysis-0"></a><a class="link" href="#timingsimplecpu-analysis-0">24.22.4.2.1. TimingSimpleCPU analysis #0</a></h6>
<div class="paragraph">
<p>Schedules <code>TimingSimpleCPU::fetch</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
TimingSimpleCPU::activateContext
SimpleThread::activate
Process::initState
ArmProcess64::initState
ArmLinuxProcess64::initState</pre>
</div>
</div>
<div class="paragraph">
<p>This schedules the initial tick, much like for for <code>AtomicSimpleCPU</code>.</p>
</div>
<div class="paragraph">
<p>This time however, it is not a tick as in <code>AtomicSimpleCPU</code>, but rather a fetch event that gets scheduled for later on, since reading DRAM memory now takes time:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>TimingSimpleCPU::activateContext(ThreadID thread_num)
{
    DPRINTF(SimpleCPU, "ActivateContext %d\n", thread_num);

    assert(thread_num &lt; numThreads);

    threadInfo[thread_num]-&gt;notIdleFraction = 1;
    if (_status == BaseSimpleCPU::Idle)
        _status = BaseSimpleCPU::Running;

    // kick things off by initiating the fetch of the next instruction
    if (!fetchEvent.scheduled())
        schedule(fetchEvent, clockEdge(Cycles(0)));</pre>
</div>
</div>
<div class="paragraph">
<p>By looking at the source, we see that <code>fetchEvent</code> runs <code>TimingSimpleCPU::fetch</code>.</p>
</div>
<div class="paragraph">
<p>Just like for <code>AtomicSimpleCPU</code>, this call comes from the <code>initState</code> call, which is exposed on <code>SimObject</code> and ultimately comes from Python.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-1"><a class="anchor" href="#timingsimplecpu-analysis-1"></a><a class="link" href="#timingsimplecpu-analysis-1">24.22.4.2.2. TimingSimpleCPU analysis #1</a></h6>
<div class="paragraph">
<p>Backtrace:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::Rank::startup
DRAMCtrl::startup</pre>
</div>
</div>
<div class="paragraph">
<p>Snippets:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
DRAMCtrl::startup()
{
    // remember the memory system mode of operation
    isTimingMode = system()-&gt;isTimingMode();

    if (isTimingMode) {
        // timestamp offset should be in clock cycles for DRAMPower
        timeStampOffset = divCeil(curTick(), tCK);

        // update the start tick for the precharge accounting to the
        // current tick
        for (auto r : ranks) {
            r-&gt;startup(curTick() + tREFI - tRP);
        }

        // shift the bus busy time sufficiently far ahead that we never
        // have to worry about negative values when computing the time for
        // the next request, this will add an insignificant bubble at the
        // start of simulation
        nextBurstAt = curTick() + tRP + tRCD;
    }
}</pre>
</div>
</div>
<div class="paragraph">
<p>which then calls:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
DRAMCtrl::Rank::startup(Tick ref_tick)
{
    assert(ref_tick &gt; curTick());

    pwrStateTick = curTick();

    // kick off the refresh, and give ourselves enough time to
    // precharge
    schedule(refreshEvent, ref_tick);
}</pre>
</div>
</div>
<div class="paragraph">
<p><code>DRAMCtrl::startup</code> is itself a <code>SimObject</code> method exposed to Python and called from <code>simulate</code> in <code>src/python/m5/simulate.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def simulate(*args, **kwargs):
    global need_startup

    if need_startup:
        root = objects.Root.getInstance()
        for obj in root.descendants(): obj.startup()</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>simulate</code> happens after <code>m5.instantiate</code>, and both are called directly from the toplevel scripts, e.g. for se.py in <code>configs/common/Simulation.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def run(options, root, testsys, cpu_class):
    ...
            exit_event = m5.simulate()</pre>
</div>
</div>
<div class="paragraph">
<p>By looking up some variable definitions in the source, we now we see some memory parameters clearly:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ranks: <code>std::vector&lt;DRAMCtrl::Rank*&gt;</code> with 2 elements. TODO why do we have 2? What does it represent? Likely linked to <a href="#gem5-config-ini"><code>config.ini</code></a> at <code>system.mem_ctrls.ranks_per_channel=2</code>: <a href="https://en.wikipedia.org/wiki/Memory_rank" class="bare">https://en.wikipedia.org/wiki/Memory_rank</a></p>
</li>
<li>
<p><code>tCK=1250</code>, <code>tREFI=7800000</code>, <code>tRP=13750</code>, <code>tRCD=13750</code>: all defined in a single code location with a comment:</p>
<div class="literalblock">
<div class="content">
<pre>     /**
     * Basic memory timing parameters initialized based on parameter
     * values.
     */</pre>
</div>
</div>
<div class="paragraph">
<p>Their values can be seen under <code>config.ini</code> and they are documented in <code>src/mem/DRAMCtrl.py</code> e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    # the base clock period of the DRAM
    tCK = Param.Latency("Clock period")

    # minimum time between a precharge and subsequent activate
    tRP = Param.Latency("Row precharge time")

    # the amount of time in nanoseconds from issuing an activate command
    # to the data being available in the row buffer for a read/write
    tRCD = Param.Latency("RAS to CAS delay")

    # refresh command interval, how often a "ref" command needs
    # to be sent. It is 7.8 us for a 64ms refresh requirement
    tREFI = Param.Latency("Refresh command interval")</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>So we realize that we are going into deep DRAM modelling, more detail that a mere mortal should ever need to know.</p>
</div>
<div class="paragraph">
<p><code>curTick() + tREFI - tRP = 0 + 7800000 - 13750 = 7786250</code> which is when that <code>refreshEvent</code> was scheduled. Our simulation ends way before that point however, so we will never know what it did thank God.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-2"><a class="anchor" href="#timingsimplecpu-analysis-2"></a><a class="link" href="#timingsimplecpu-analysis-2">24.22.4.2.3. TimingSimpleCPU analysis #2</a></h6>
<div class="paragraph">
<p>This is just the startup of the second rank, see: <a href="#timingsimplecpu-analysis-1">TimingSimpleCPU analysis #1</a>.</p>
</div>
<div class="paragraph">
<p><code>se.py</code> allocates the memory controller at <code>configs/common/MemConfig.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>def config_mem(options, system):

    ...

    opt_mem_channels = options.mem_channels

    ...

    nbr_mem_ctrls = opt_mem_channels

    ...

    for r in system.mem_ranges:
        for i in range(nbr_mem_ctrls):
            mem_ctrl = create_mem_ctrl(cls, r, i, nbr_mem_ctrls, intlv_bits,
                                       intlv_size)

            ...

            mem_ctrls.append(mem_ctrl)</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-3-and-4"><a class="anchor" href="#timingsimplecpu-analysis-3-and-4"></a><a class="link" href="#timingsimplecpu-analysis-3-and-4">24.22.4.2.4. TimingSimpleCPU analysis #3 and #4</a></h6>
<div class="paragraph">
<p>From the timing we know what that one is: the end of time exit event, like for <code>AtomicSimpleCPU</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-5"><a class="anchor" href="#timingsimplecpu-analysis-5"></a><a class="link" href="#timingsimplecpu-analysis-5">24.22.4.2.5. TimingSimpleCPU analysis #5</a></h6>
<div class="paragraph">
<p>Executes <code>TimingSimpleCPU::fetch()</code>.</p>
</div>
<div class="paragraph">
<p>The log shows that event ID <code>43</code> is now executing: we had previously seen event <code>43</code> get scheduled and had analyzed it to be the initial fetch.</p>
</div>
<div class="paragraph">
<p>We can step into <code>TimingSimpleCPU::fetch()</code> to confirm that the expected <a href="#elf">ELF</a> entry point is being fetched. We can inspect the ELF with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch aarch64 readelf -- \
  -h "$(./getvar --arch aarch64 userland_build_dir)/arch/aarch64/freestanding/linux/hello.out"</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  Entry point address:               0x400078</pre>
</div>
</div>
<div class="paragraph">
<p>and by the time we go past:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>TimingSimpleCPU::fetch()
{
    ...
    if (needToFetch) {
        ...
        setupFetchRequest(ifetch_req);
        DPRINTF(SimpleCPU, "Translating address %#x\n", ifetch_req-&gt;getVaddr());
        thread-&gt;itb-&gt;translateTiming(ifetch_req, thread-&gt;getTC(),
                &amp;fetchTranslation, BaseTLB::Execute);</pre>
</div>
</div>
<div class="paragraph">
<p><code>BaseSimpleCPU::setupFetchRequest</code> sets up the fetch of the expected entry point by reading the PC:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p/x ifetch_req-&gt;getVaddr()</pre>
</div>
</div>
<div class="paragraph">
<p>Still during the execution of the <code>fetch</code>, execution then moves into the address translation <code>ArmISA::TLB::translateTiming</code>, and after a call to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>TLB::translateSe</pre>
</div>
</div>
<div class="paragraph">
<p>the packet now contains the physical address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>_paddr = 0x78</pre>
</div>
</div>
<div class="paragraph">
<p>so we deduce that the virtual address 0x400078 maps to the physical address 0x78. But of course, <a href="https://lmgtfy.com/">let me log that for you</a> by adding <code>--trace MMU</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: MMU: system.cpu.workload: Translating: 0x400078-&gt;0x78</pre>
</div>
</div>
<div class="paragraph">
<p>If we try <code>--trace DRAM</code> we can see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>      0: DRAM: system.mem_ctrls: recvTimingReq: request ReadReq addr 120 size 4</pre>
</div>
</div>
<div class="paragraph">
<p>where 120 == 0x78 (it logs addresses in decimal? Really??) and the size 4 which is the instruction width.</p>
</div>
<div class="paragraph">
<p>Now that we are here, we might as well learn how to log the data that was fetched from DRAM.</p>
</div>
<div class="paragraph">
<p>Fist we determine the expected bytes from the <a href="#disas">disassembly</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --userland userland/arch/aarch64/freestanding/linux/hello.S _start</pre>
</div>
</div>
<div class="paragraph">
<p>which shows us the initial instruction encodings near the entry point <code>_start</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x0000000000400078 &lt;+0&gt;:     20 00 80 d2     mov     x0, #0x1                        // #1
   0x000000000040007c &lt;+4&gt;:     e1 00 00 10     adr     x1, 0x400098 &lt;msg&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>Now, TODO :-) The <code>DRAM</code> logs don&#8217;t contain data. Maybe this can be done with <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/comm_monitor.hh#L55"><code>CommMonitor</code></a>, but it is no exposed on fs.py</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-6"><a class="anchor" href="#timingsimplecpu-analysis-6"></a><a class="link" href="#timingsimplecpu-analysis-6">24.22.4.2.6. TimingSimpleCPU analysis #6</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::processNextReqEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::addToReadQueue
DRAMCtrl::recvTimingReq
DRAMCtrl::MemoryPort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
CoherentXBar::recvTimingReq
CoherentXBar::CoherentXBarSlavePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch</pre>
</div>
</div>
<div class="paragraph">
<p>The event loop has started, and magic initialization schedulings are not happening anymore: now every event is being scheduled from another event:</p>
</div>
<div class="paragraph">
<p>From the trace, we see that we are already running from the event queue under <code>TimingSimpleCPU::fetch</code> as expected.</p>
</div>
<div class="paragraph">
<p>From the backtrace we see the tortuous path that the data request takes, going through:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::TLB</code></p>
</li>
<li>
<p><code>CoherentXBar</code></p>
</li>
<li>
<p><code>DRAMCtrl</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This matches the <code>config.ini</code> system image, since we see that the request goes through the <code>CoherentXBar</code> before reaching memory, like all other CPU memory accesses, see also: <a href="#gem5-crossbar-interconnect">gem5 crossbar interconnect</a>.</p>
</div>
<div class="paragraph">
<p>The scheduling happens at frame <code>DRAMCtrl::addToReadQueue</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>     // If we are not already scheduled to get a request out of the
     // queue, do so now
     if (!nextReqEvent.scheduled()) {
         DPRINTF(DRAM, "Request scheduled immediately\n");
         schedule(nextReqEvent, curTick());
     }</pre>
</div>
</div>
<div class="paragraph">
<p>From this we deduce that the DRAM has a request queue of some sort, and that the fetch:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>has added a read request to that queue</p>
</li>
<li>
<p>and has made a future request to read from the queue</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The signature of the function is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>DRAMCtrl::addToReadQueue(PacketPtr pkt, unsigned int pktCount)</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>PacketPtr</code> is of <code>class `Packet</code>, and so clearly the packet is coming from above.</p>
</div>
<div class="paragraph">
<p>From:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>p/x *pkt</pre>
</div>
</div>
<div class="paragraph">
<p>we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>addr = 0x78</pre>
</div>
</div>
<div class="paragraph">
<p>which from <a href="#timingsimplecpu-analysis-5">TimingSimpleCPU analysis #5</a> we know is the physical address of the ELF entry point.</p>
</div>
<div class="paragraph">
<p>Communication goes through certain components via the <code>class Port</code> interface, e.g. at <code>TimingSimpleCPU::sendFetch</code> a call is made to send the packet forward:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>icachePort.sendTimingReq(ifetch_pkt)</pre>
</div>
</div>
<div class="paragraph">
<p>which ends up calling:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>peer-&gt;recvTimingReq(pkt);</pre>
</div>
</div>
<div class="paragraph">
<p>to reach the receiving side:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CoherentXBar::CoherentXBarSlavePort::recvTimingReq</pre>
</div>
</div>
<div class="paragraph">
<p>Ports are also used to connect the XBar and the DRAM.</p>
</div>
<div class="paragraph">
<p>We will then see that at <a href="#timingsimplecpu-analysis-20">TimingSimpleCPU analysis #20</a> a reply packet will come back through the port interface down to the icache port, and only then does the decoding and execution happen.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-7"><a class="anchor" href="#timingsimplecpu-analysis-7"></a><a class="link" href="#timingsimplecpu-analysis-7">24.22.4.2.7. TimingSimpleCPU analysis #7</a></h6>
<div class="paragraph">
<p>Schedules <code>BaseXBar::Layer::releaseLayer</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
BaseXBar::Layer&lt;SlavePort, MasterPort&gt;::occupyLayer
BaseXBar::Layer&lt;SlavePort, MasterPort&gt;::succeededTiming
CoherentXBar::recvTimingReq
CoherentXBar::CoherentXBarSlavePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch</pre>
</div>
</div>
<div class="paragraph">
<p>which schedules a <code>SimpleMemory::release</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-8"><a class="anchor" href="#timingsimplecpu-analysis-8"></a><a class="link" href="#timingsimplecpu-analysis-8">24.22.4.2.8. TimingSimpleCPU analysis #8</a></h6>
<div class="paragraph">
<p>Executes <code>DRAMCtrl::processNextReqEvent</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-9"><a class="anchor" href="#timingsimplecpu-analysis-9"></a><a class="link" href="#timingsimplecpu-analysis-9">24.22.4.2.9. TimingSimpleCPU analysis #9</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::Rank::processActivateEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::activateBank
DRAMCtrl::doDRAMAccess
DRAMCtrl::processNextReqEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-10"><a class="anchor" href="#timingsimplecpu-analysis-10"></a><a class="link" href="#timingsimplecpu-analysis-10">24.22.4.2.10. TimingSimpleCPU analysis #10</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::processRespondEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::processNextReqEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-11"><a class="anchor" href="#timingsimplecpu-analysis-11"></a><a class="link" href="#timingsimplecpu-analysis-11">24.22.4.2.11. TimingSimpleCPU analysis #11</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::processNextReqEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::processNextReqEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-12"><a class="anchor" href="#timingsimplecpu-analysis-12"></a><a class="link" href="#timingsimplecpu-analysis-12">24.22.4.2.12. TimingSimpleCPU analysis #12</a></h6>
<div class="paragraph">
<p>Executes <code>DRAMCtrl::Rank::processActivateEvent</code>.</p>
</div>
<div class="paragraph">
<p>which schedules:</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-13"><a class="anchor" href="#timingsimplecpu-analysis-13"></a><a class="link" href="#timingsimplecpu-analysis-13">24.22.4.2.13. TimingSimpleCPU analysis #13</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::Rank::processPowerEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::Rank::schedulePowerEvent
DRAMCtrl::Rank::processActivateEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-14"><a class="anchor" href="#timingsimplecpu-analysis-14"></a><a class="link" href="#timingsimplecpu-analysis-14">24.22.4.2.14. TimingSimpleCPU analysis #14</a></h6>
<div class="paragraph">
<p>Executes <code>DRAMCtrl::Rank::processPowerEvent</code>.</p>
</div>
<div class="paragraph">
<p>This it must just be some power statistics stuff, as it does not schedule anything else.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-15"><a class="anchor" href="#timingsimplecpu-analysis-15"></a><a class="link" href="#timingsimplecpu-analysis-15">24.22.4.2.15. TimingSimpleCPU analysis #15</a></h6>
<div class="paragraph">
<p>Executes <code>BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-16"><a class="anchor" href="#timingsimplecpu-analysis-16"></a><a class="link" href="#timingsimplecpu-analysis-16">24.22.4.2.16. TimingSimpleCPU analysis #16</a></h6>
<div class="paragraph">
<p>Executes <code>DRAMCtrl::processNextReqEvent()</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-17"><a class="anchor" href="#timingsimplecpu-analysis-17"></a><a class="link" href="#timingsimplecpu-analysis-17">24.22.4.2.17. TimingSimpleCPU analysis #17</a></h6>
<div class="paragraph">
<p>Executes <code>DRAMCtrl::processRespondEvent()</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-18"><a class="anchor" href="#timingsimplecpu-analysis-18"></a><a class="link" href="#timingsimplecpu-analysis-18">24.22.4.2.18. TimingSimpleCPU analysis #18</a></h6>
<div class="paragraph">
<p>Schedules <code>PacketQueue::processSendEvent()</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>PacketQueue::schedSendEvent
PacketQueue::schedSendTiming
QueuedSlavePort::schedTimingResp
DRAMCtrl::accessAndRespond
DRAMCtrl::processRespondEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-19"><a class="anchor" href="#timingsimplecpu-analysis-19"></a><a class="link" href="#timingsimplecpu-analysis-19">24.22.4.2.19. TimingSimpleCPU analysis #19</a></h6>
<div class="paragraph">
<p>Executes <code>PacketQueue::processSendEvent()</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-20"><a class="anchor" href="#timingsimplecpu-analysis-20"></a><a class="link" href="#timingsimplecpu-analysis-20">24.22.4.2.20. TimingSimpleCPU analysis #20</a></h6>
<div class="paragraph">
<p>Schedules <code>PacketQueue::processSendEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
PacketQueue::schedSendEvent
PacketQueue::schedSendTiming
QueuedSlavePort::schedTimingResp
CoherentXBar::recvTimingResp
CoherentXBar::CoherentXBarMasterPort::recvTimingResp
TimingResponseProtocol::sendResp
SlavePort::sendTimingResp
RespPacketQueue::sendTiming
PacketQueue::sendDeferredPacket
PacketQueue::processSendEvent</pre>
</div>
</div>
<div class="paragraph">
<p>From this backtrace, we see that this event is happening as the fetch reply packet finally comes back from DRAM.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-21"><a class="anchor" href="#timingsimplecpu-analysis-21"></a><a class="link" href="#timingsimplecpu-analysis-21">24.22.4.2.21. TimingSimpleCPU analysis #21</a></h6>
<div class="paragraph">
<p>Schedules <code>BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
BaseXBar::Layer&lt;MasterPort, SlavePort&gt;::occupyLayer
BaseXBar::Layer&lt;MasterPort, SlavePort&gt;::succeededTiming
CoherentXBar::recvTimingResp
CoherentXBar::CoherentXBarMasterPort::recvTimingResp
TimingResponseProtocol::sendResp
SlavePort::sendTimingResp
RespPacketQueue::sendTiming
PacketQueue::sendDeferredPacket
PacketQueue::processSendEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-22"><a class="anchor" href="#timingsimplecpu-analysis-22"></a><a class="link" href="#timingsimplecpu-analysis-22">24.22.4.2.22. TimingSimpleCPU analysis #22</a></h6>
<div class="paragraph">
<p>Executes <code>BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-23"><a class="anchor" href="#timingsimplecpu-analysis-23"></a><a class="link" href="#timingsimplecpu-analysis-23">24.22.4.2.23. TimingSimpleCPU analysis #23</a></h6>
<div class="paragraph">
<p>Executes <code>PacketQueue::processSendEvent</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-24"><a class="anchor" href="#timingsimplecpu-analysis-24"></a><a class="link" href="#timingsimplecpu-analysis-24">24.22.4.2.24. TimingSimpleCPU analysis #24</a></h6>
<div class="paragraph">
<p>Schedules <code>TimingSimpleCPU::IcachePort::ITickEvent::process()</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
TimingSimpleCPU::TimingCPUPort::TickEvent::schedule
TimingSimpleCPU::IcachePort::recvTimingResp
TimingResponseProtocol::sendResp
SlavePort::sendTimingResp
RespPacketQueue::sendTiming
PacketQueue::sendDeferredPacket
PacketQueue::processSendEvent</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-25"><a class="anchor" href="#timingsimplecpu-analysis-25"></a><a class="link" href="#timingsimplecpu-analysis-25">24.22.4.2.25. TimingSimpleCPU analysis #25</a></h6>
<div class="paragraph">
<p>Executes <code>TimingSimpleCPU::IcachePort::ITickEvent::process()</code>.</p>
</div>
<div class="paragraph">
<p>This custom <code>process</code> then calls <code>TimingSimpleCPU::completeIfetch(PacketPtr pkt)</code>, and that finally executes the very first instruction:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>77000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)</pre>
</div>
</div>
<div class="paragraph">
<p>The end of this instruction must be setting things up in a way that can continue the PC walk loop, and by looking at the source and traces, it is clearly from: <code>TimingSimpleCPU::advanceInst</code> which calls <code>TimingSimpleCPU::fetch</code>.</p>
</div>
<div class="paragraph">
<p>And <code>TimingSimpleCPU::fetch</code> is the very thing we did in this simulation at <a href="#timingsimplecpu-analysis-0">TimingSimpleCPU analysis #0</a>!!! OMG, that&#8217;s the loop.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-26"><a class="anchor" href="#timingsimplecpu-analysis-26"></a><a class="link" href="#timingsimplecpu-analysis-26">24.22.4.2.26. TimingSimpleCPU analysis #26</a></h6>
<div class="paragraph">
<p>Schedules <code>DRAMCtrl::processNextReqEvent</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
DRAMCtrl::addToReadQueue
DRAMCtrl::recvTimingReq
DRAMCtrl::MemoryPort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
CoherentXBar::recvTimingReq
CoherentXBar::CoherentXBarSlavePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch
TimingSimpleCPU::advanceInst
TimingSimpleCPU::completeIfetch
TimingSimpleCPU::IcachePort::ITickEvent::process</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-27"><a class="anchor" href="#timingsimplecpu-analysis-27"></a><a class="link" href="#timingsimplecpu-analysis-27">24.22.4.2.27. TimingSimpleCPU analysis #27</a></h6>
<div class="paragraph">
<p>Schedules <code>BaseXBar::Layer&lt;SrcType, DstType&gt;::releaseLayer</code> through:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
BaseXBar::Layer&lt;SlavePort, MasterPort&gt;::occupyLayer
BaseXBar::Layer&lt;SlavePort, MasterPort&gt;::succeededTiming
CoherentXBar::recvTimingReq
CoherentXBar::CoherentXBarSlavePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch
TimingSimpleCPU::advanceInst
TimingSimpleCPU::completeIfetch
TimingSimpleCPU::IcachePort::ITickEvent::process</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-28"><a class="anchor" href="#timingsimplecpu-analysis-28"></a><a class="link" href="#timingsimplecpu-analysis-28">24.22.4.2.28. TimingSimpleCPU analysis #28</a></h6>
<div class="paragraph">
<p>Execute <code>DRAMCtrl::processNextReqEvent</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-29"><a class="anchor" href="#timingsimplecpu-analysis-29"></a><a class="link" href="#timingsimplecpu-analysis-29">24.22.4.2.29. TimingSimpleCPU analysis #29</a></h6>
<div class="paragraph">
<p>Schedule <code>DRAMCtrl::processRespondEvent()</code>.</p>
</div>
</div>
<div class="sect5">
<h6 id="timingsimplecpu-analysis-ldr-stall"><a class="anchor" href="#timingsimplecpu-analysis-ldr-stall"></a><a class="link" href="#timingsimplecpu-analysis-ldr-stall">24.22.4.2.30. TimingSimpleCPU analysis: LDR stall</a></h6>
<div class="paragraph">
<p>One important thing we want to check now, is how the memory reads are going to make the processor stall in the middle of an instruction.</p>
</div>
<div class="paragraph">
<p>This is also discussed at: <a href="#gem5-execute-vs-initiateacc-vs-completeacc">gem5 <code>execute</code> vs <code>initiateAcc</code> vs <code>completeAcc</code></a>.</p>
</div>
<div class="paragraph">
<p>Since we were using a simple CPU without a pipeline, the data memory access stall everything: there is no further progress until memory comes back.</p>
</div>
<div class="paragraph">
<p>For that, we can GDB to the <code>TimingSimpleCPU::completeIfetch</code> of the first LDR done in our test program.</p>
</div>
<div class="paragraph">
<p>By doing that, we see that this time at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if (curStaticInst &amp;&amp; curStaticInst-&gt;isMemRef()) {
    // load or store: just send to dcache
    Fault fault = curStaticInst-&gt;initiateAcc(&amp;t_info, traceData);

    if (_status == BaseSimpleCPU::Running) {
    }
} else if (curStaticInst) {
    // non-memory instruction: execute completely now
    Fault fault = curStaticInst-&gt;execute(&amp;t_info, traceData);</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><code>curStaticInst-&gt;isMemRef()</code> is true, and there is no instruction <code>execute</code> call in that part of the branch, only for instructions that don&#8217;t touch memory</p>
</li>
<li>
<p><code>_status</code> is <code>BaseSimpleCPU::Status::DcacheWaitResponse</code> and <code>advanceInst</code> is not yet called</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We can verify that <code>execute</code> never happens by putting a breakpoint on <code>ArmISAInst::LDRXL64_LIT::execute</code> which never gets called.</p>
</div>
<div class="paragraph">
<p>Therefore, we conclude that <code>initiateAcc</code> is what actually starts the memory request.</p>
</div>
<div class="paragraph">
<p>Later on, when the memory access completes the event calls <code>TimingSimpleCPU::completeDataAccess</code> which calls <code>ArmISAInst::LDRXL64_LIT::completeAcc</code>, which sets the register value to what was read from memory.</p>
</div>
<div class="paragraph">
<p>More memory event details can be seen at: <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">gem5 functional vs atomic vs timing memory requests</a>.</p>
</div>
<div class="paragraph">
<p>The following is the region of interest of the event log:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 175000: Event: Event_40: Timing CPU icache tick 40 executed @ 175000
 175000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 175000
 175000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 scheduled @ 176000

 175000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 175000
 175000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 scheduled @ 193750
 175000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 175000

 175000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 175000

 176000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 60 executed @ 176000

 193750: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 executed @ 193750
 193750: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 scheduled @ 221750

 221750: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 executed @ 221750
 221750: Event: system.membus.slave[2]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 66 scheduled @ 224000
 221750: Event: system.membus.respLayer2.wrapped_function_event: EventFunctionWrapped 67 scheduled @ 224000

 224000: Event: system.membus.respLayer2.wrapped_function_event: EventFunctionWrapped 67 executed @ 224000

 224000: Event: system.membus.slave[2]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 66 executed @ 224000
 224000: Event: Event_42: Timing CPU dcache tick 42 scheduled @ 224000

 224000: Event: Event_42: Timing CPU dcache tick 42 executed @ 224000
 175000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>We first find it by looking for the <code>ExecEnable</code> of LDR.</p>
</div>
<div class="paragraph">
<p>Then, we go up to the previous <code>Timing CPU icache tick</code> event, which from the analysis of previous instruction traces, we know is where the instruction execution starts, the LDR instruction fetch is done by then!</p>
</div>
<div class="paragraph">
<p>Next, several events happen as the data request must be percolating through the memory system, it must be very similar to the instruction fetches. TODO analyze event function names.</p>
</div>
<div class="paragraph">
<p>Finally, at last we reach</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 224000: Event: Event_42: Timing CPU dcache tick 42 executed @ 224000
 175000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   ldr   w2, #4194464       : MemRead :  D=0x0000000000000006 A=0x4000a0  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>from which we guess:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>224000</code>: this is the time that the data request finally returned, and at which execute gets called</p>
</li>
<li>
<p><code>175000</code>: the log finally prints at the end of execution, but it does not show the actual time that things finished, but rather the time that the ifetch finished, which happened in the past</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches"><a class="anchor" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches"></a><a class="link" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">24.22.4.3. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a></h5>
<div class="paragraph">
<p>Let&#8217;s just add <code>--caches</code> to <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a> to see if things go any faster, and add <code>Cache</code> to <code>--trace</code> as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--trace Cache,Event,ExecAll,-ExecSymbol,FmtFlag</pre>
</div>
</div>
<div class="paragraph">
<p>The resulting trace is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#0         0: Event: system.cpu.wrapped_function_event: EventFunctionWrapped 43 scheduled @ 0
#2         0: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 14 scheduled @ 7786250
#3         0: Event: system.mem_ctrls_1.wrapped_function_event: EventFunctionWrapped 20 scheduled @ 7786250
#4         0: Event: Event_84: generic 84 scheduled @ 0
#5         0: Event: Event_84: generic 84 rescheduled @ 18446744073709551615
#6         0: Event: system.cpu.wrapped_function_event: EventFunctionWrapped 43 executed @ 0
#7         0: Cache: system.cpu.icache: access for ReadReq [78:7b] IF miss
#8         0: Event: system.cpu.icache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 59 scheduled @ 1000
#9      1000: Event: system.cpu.icache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 59 executed @ 1000
#10     1000: Cache: system.cpu.icache: sendMSHRQueuePacket: MSHR ReadReq [78:7b] IF
#12     1000: Cache: system.cpu.icache: createMissPacket: created ReadCleanReq [40:7f] IF from ReadReq [78:7b] IF
#13     1000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 1000
#14     1000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 70 scheduled @ 2000
#15     1000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 1000
#16     1000: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 12 scheduled @ 1000
#17     1000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 scheduled @ 46250
#18     1000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 scheduled @ 5000
#19     1000: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 12 executed @ 1000
#20     1000: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 15 scheduled @ 1000
#22     1000: Event: system.mem_ctrls_0.wrapped_function_event: EventFunctionWrapped 15 executed @ 1000
#23     2000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 70 executed @ 2000
#24     5000: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 9 executed @ 5000
#25    46250: Event: system.mem_ctrls.wrapped_function_event: EventFunctionWrapped 10 executed @ 46250
#26    46250: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 scheduled @ 74250
#27    74250: Event: system.mem_ctrls.port-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 8 executed @ 74250
#28    74250: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 74 scheduled @ 77000
#29    74250: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 75 scheduled @ 80000
#30    77000: Event: system.membus.slave[1]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 74 executed @ 77000
#32    77000: Cache: system.cpu.icache: recvTimingResp: Handling response ReadResp [40:7f] IF
#33    77000: Cache: system.cpu.icache: Block for addr 0x40 being updated in Cache
#34    77000: Cache: system.cpu.icache: Block addr 0x40 (ns) moving from state 0 to state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x1 way: 0
#35    77000: Event: system.cpu.icache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 57 scheduled @ 78000
#36    78000: Event: system.cpu.icache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 57 executed @ 78000
#37    78000: Event: Event_40: Timing CPU icache tick 40 scheduled @ 78000
#38    78000: Event: Event_40: Timing CPU icache tick 40 executed @ 78000
#39    78000: ExecEnable: system.cpu: A0 T0 : 0x400078    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
#40    78000: Cache: system.cpu.icache: access for ReadReq [7c:7f] IF hit state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x1 way: 0
#42    78000: Event: system.cpu.icache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 57 scheduled @ 83000
#43    80000: Event: system.membus.respLayer1.wrapped_function_event: EventFunctionWrapped 75 executed @ 80000
#44    83000: Event: system.cpu.icache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 57 executed @ 83000
#45    83000: Event: Event_40: Timing CPU icache tick 40 scheduled @ 83000
#46    83000: Event: Event_40: Timing CPU icache tick 40 executed @ 83000
#47    83000: ExecEnable: system.cpu: A0 T0 : 0x40007c    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)
#48    83000: Event: system.cpu.icache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 59 scheduled @ 84000
          [...]
      191000: Event: Event_85: generic 85 scheduled @ 191000
      191000: Event: Event_85: generic 85 executed @ 191000</pre>
</div>
</div>
<div class="paragraph">
<p>So yes, <code>--caches</code> does work here, leading to a runtime of 191000 rather than 469000 without caches!</p>
</div>
<div class="paragraph">
<p>Notably, we now see that very little time passed between the first and second instructions which are marked with <code>ExecEnable</code> in #39 and #47, presumably because rather than going out all the way to the DRAM system the event chain stops right at the <code>icache.cpu_side</code> when a hit happens, which must have been the case for the second instruction, which is just adjacent to the first one.</p>
</div>
<div class="paragraph">
<p>It is also interested to look into the generated <a href="#gem5-config-ini"><code>config.dot.svg</code></a> to compare it to the one without caches: <a href="#config-dot-svg-timingsimplecpu">Figure 3, &#8220;<code>config.dot.svg</code> for a TimingSimpleCPU without caches.&#8221;</a>. With caches: <a href="#config-dot-svg-timingsimplecpu-caches">Figure 4, &#8220;<code>config.dot.svg</code> for a TimingSimpleCPU with caches.&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>We can see from there, that we now have <code>icache</code> and <code>dcache</code> elements inside the CPU block, and that the CPU <code>icache</code> and <code>dcache</code> ports go through the caches to the <code>SystemXBar</code> rather than being directly connected as before.</p>
</div>
<div class="paragraph">
<p>It is worth noting that the caches do not affect the <code>ArmITB</code> and <code>ArmDTB</code> <a href="#arm-paging">TLBs</a>, since those are already caches themselves.</p>
</div>
<div id="config-dot-svg-timingsimplecpu-caches" class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/media/master/gem5_config_TimingSimpleCPU_caches_12c917de54145d2d50260035ba7fa614e25317a3.svg?sanitize=true" alt="gem5 config TimingSimpleCPU caches 12c917de54145d2d50260035ba7fa614e25317a3" height="600">
</div>
<div class="title">Figure 4. <code>config.dot.svg</code> for a TimingSimpleCPU with caches.</div>
</div>
<div class="paragraph">
<p>We can break down the events between the instructions as follows.</p>
</div>
<div class="paragraph">
<p>First, based on <a href="#timingsimplecpu-analysis-5">TimingSimpleCPU analysis #5</a>, we <code>b TimingSimpleCPU::fetch</code> to see how the initial magically scheduled fetch, and necessarily cache miss, work:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
PacketQueue::schedSendEvent
BaseCache::CacheMasterPort::schedSendEvent
BaseCache::schedMemSideSendEvent
BaseCache::allocateMissBuffer
BaseCache::handleTimingReqMiss
Cache::handleTimingReqMiss
BaseCache::recvTimingReq
Cache::recvTimingReq
BaseCache::CpuSidePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch</pre>
</div>
</div>
<div class="paragraph">
<p>By comparing this to the uncached access at <a href="#timingsimplecpu-analysis-25">TimingSimpleCPU analysis #25</a>, we see that this one does not reach the <code>CoherentXBar</code> at all: the cache must be scheduling an event in the future to model a delay between the cache request and XBar communication.</p>
</div>
<div class="paragraph">
<p>A quick source structural view shows that the source for <a href="#gem5-ruby-build">non-Ruby caches</a> such as the ones from this example are located under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/mem/cache</pre>
</div>
</div>
<div class="paragraph">
<p>and the following simple class hierarchy:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>BaseCache</code></p>
<div class="ulist">
<ul>
<li>
<p><code>Cache</code></p>
</li>
<li>
<p><code>NoncoherentCache</code></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Next, we fast forward to <code>#39</code> with <code>b TimingSimpleCPU::IcachePort::ITickEvent::process</code> which as we knows from previous sections, is the event that executes instructions, and therefore leaves us at the start of the second instruction.</p>
</div>
<div class="paragraph">
<p>Then, we <code>b EventManager::schedule</code> to see what that schedules:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>EventManager::schedule
PacketQueue::schedSendEvent
PacketQueue::schedSendTiming
QueuedSlavePort::schedTimingResp
BaseCache::handleTimingReqHit
Cache::handleTimingReqHit
BaseCache::recvTimingReq
Cache::recvTimingReq
BaseCache::CpuSidePort::recvTimingReq
TimingRequestProtocol::sendReq
MasterPort::sendTimingReq
TimingSimpleCPU::sendFetch
TimingSimpleCPU::FetchTranslation::finish
ArmISA::TLB::translateComplete
ArmISA::TLB::translateTiming
ArmISA::TLB::translateTiming
TimingSimpleCPU::fetch
TimingSimpleCPU::advanceInst
TimingSimpleCPU::completeIfetch
TimingSimpleCPU::IcachePort::ITickEvent::process</pre>
</div>
</div>
<div class="paragraph">
<p>By comparing this trace from the this cache hit and the previous cache miss, we see that <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/base.cc#L337"><code>BaseCache::recvTimingReq</code></a> decides between either: <code>Cache::handleTimingReqHit</code> and <code>Cache::handleTimingReqMiss</code>, and from there we see that the key function that decides if the block is present is <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/base.cc#L1033"><code>BaseCache::access</code></a>.</p>
</div>
<div class="paragraph">
<p>We can see access behaviour at on the log lines, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#7         0: Cache: system.cpu.icache: access for ReadReq [78:7b] IF miss
#40    78000: Cache: system.cpu.icache: access for ReadReq [7c:7f] IF hit state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x1 way: 0</pre>
</div>
</div>
<div class="paragraph">
<p>which makes sense since from <a href="#timingsimplecpu-analysis-5">TimingSimpleCPU analysis #5</a> we know that the physical address of the initial instruction is 0x78, and 4 bytes are read for each instruction, so the second instruction access is at 0x7c.</p>
</div>
<div class="paragraph">
<p>The hit line also shows the precise cache state <code>E</code> from the MOESI protocol: <a href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5">What is the coherency protocol implemented by the classic cache system in gem5?</a>.</p>
</div>
<div class="paragraph">
<p>The other log lines are also very clear, e.g. for the miss we see the following lines:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#10     1000: Cache: system.cpu.icache: sendMSHRQueuePacket: MSHR ReadReq [78:7b] IF
#12     1000: Cache: system.cpu.icache: createMissPacket: created ReadCleanReq [40:7f] IF from ReadReq [78:7b] IF
#32    77000: Cache: system.cpu.icache: recvTimingResp: Handling response ReadResp [40:7f] IF
#33    77000: Cache: system.cpu.icache: Block for addr 0x40 being updated in Cache
#34    77000: Cache: system.cpu.icache: Block addr 0x40 (ns) moving from state 0 to state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x1 way: 0</pre>
</div>
</div>
<div class="paragraph">
<p>This shows us that the cache miss fills the cache line 40:7f, so we deduce that the cache block size is 0x40 == 64 bytes. The second address only barely hit at the last bytes of the block!</p>
</div>
<div class="paragraph">
<p>It also informs us that the cache moved to <code>E</code> (from the initial <code>I</code>) state since a memory read was done.</p>
</div>
<div class="paragraph">
<p>We can confirm this with <code>--trace DRAM</code> which shows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   1000: DRAM: system.mem_ctrls: recvTimingReq: request ReadCleanReq addr 64 size 64</pre>
</div>
</div>
<div class="paragraph">
<p>Contrast this with the non <code>--cache</code> version seen at <a href="#timingsimplecpu-analysis-5">TimingSimpleCPU analysis #5</a> in which DRAM only actually reads the 4 required bytes.</p>
</div>
<div class="paragraph">
<p>The only cryptic thing about the messages is the <code>IF</code> flag, but good computer architects would have guessed it correctly that it is "instruction fetch" and <a href="https://github.com/gem5/gem5/blob/fa70478413e4650d0058cbfe81fd5ce362101994/src/mem/packet.cc#L372">src/mem/packet.cc</a> confirms:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>void
Packet::print(std::ostream &amp;o, const int verbosity,
              const std::string &amp;prefix) const
{
    ccprintf(o, "%s%s [%x:%x]%s%s%s%s%s%s", prefix, cmdString(),
             getAddr(), getAddr() + getSize() - 1,
             req-&gt;isSecure() ? " (s)" : "",
             req-&gt;isInstFetch() ? " IF" : "",
             req-&gt;isUncacheable() ? " UC" : "",
             isExpressSnoop() ? " ES" : "",
             req-&gt;isToPOC() ? " PoC" : "",
             req-&gt;isToPOU() ? " PoU" : "");
}</pre>
</div>
</div>
<div class="paragraph">
<p>Another interesting observation of running with <code>--trace Cache,DRAM,XBar</code> is that between the execution of both instructions, there is a <code>Cache</code> event, but no <code>DRAM</code> or <code>XBar</code> events:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  78000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  flags=(IsInteger)
  78000: Cache: system.cpu.icache: access for ReadReq [7c:7f] IF hit state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x1 way: 0
  83000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  flags=(IsInteger)</pre>
</div>
</div>
<div class="paragraph">
<p>which is further consistent with the cache hit idea: no traffic goes down to the DRAM nor crossbar.</p>
</div>
<div class="paragraph">
<p>This block size parameter can be seen set on the <a href="#gem5-config-ini">gem5 config.ini</a> file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system]
cache_line_size=64</pre>
</div>
</div>
<div class="paragraph">
<p>so it is runtime configurable. The other key cache parameters can be seen further down in the config:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.dcache]
assoc=2
size=65536

[system.cpu.dcache.replacement_policy]
type=LRURP

[system.cpu.dcache.tags.indexing_policy]
type=SetAssociative</pre>
</div>
</div>
<div class="paragraph">
<p>so we understand that by default the classic cache:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>is 2-way <a href="https://en.wikipedia.org/wiki/CPU_cache#Two-way_set_associative_cache" class="bare">https://en.wikipedia.org/wiki/CPU_cache#Two-way_set_associative_cache</a></p>
</li>
<li>
<p>has 16KiB total size</p>
</li>
<li>
<p>uses LRURP <a href="https://en.wikipedia.org/wiki/Cache_replacement_policies">replacement policy</a>. LRU is a well known policy, "LRU RP" seems to simply stand for "LRU Replacement Policy". Other policies can be seen under: <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/replacement_policies/">src/mem/cache/replacement_policies/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>At:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#7         0: Cache: system.cpu.icache: access for ReadReq [78:7b] IF miss
#8         0: Event: system.cpu.icache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 59 scheduled @ 1000
#9      1000: Event: system.cpu.icache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 59 executed @ 1000
#10     1000: Cache: system.cpu.icache: sendMSHRQueuePacket: MSHR ReadReq [78:7b] IF
#12     1000: Cache: system.cpu.icache: createMissPacket: created ReadCleanReq [40:7f] IF from ReadReq [78:7b] IF</pre>
</div>
</div>
<div class="paragraph">
<p>we can briefly see the <a href="#gem5-mshr">gem5 <code>MSHR</code></a> doing its thing.</p>
</div>
<div class="paragraph">
<p>At time 0, the CPU icache wants to read, so it creates a <a href="#gem5-packet">packet</a> that reads 4 bytes only (<code>[78:7b]</code>) for the instruction, and that goes into the MSHR, to be treated in a future event.</p>
</div>
<div class="paragraph">
<p>At 1000, the future event is executed, and so it reads the original packet from the MSHR, and uses that to create a new request <code>[40:7f]</code> which gets forwarded.</p>
</div>
<div class="sect5">
<h6 id="what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5"><a class="anchor" href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5"></a><a class="link" href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5">24.22.4.3.1. What is the coherency protocol implemented by the classic cache system in gem5?</a></h6>
<div class="paragraph">
<p><a href="#moesi">MOESI cache coherence protocol</a>: <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L352" class="bare">https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L352</a></p>
</div>
<div class="paragraph">
<p>The actual representation is done via separate state bits: <a href="https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L66" class="bare">https://github.com/gem5/gem5/blob/9fc9c67b4242c03f165951775be5cd0812f2a705/src/mem/cache/cache_blk.hh#L66</a> and MOESI appears explicitly only on the pretty printing.</p>
</div>
<div class="paragraph">
<p>This pretty printing appears for example in the <code>--trace Cache</code> lines as shown at <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a> and with a few more transitions visible at <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">Section 24.22.4.4, &#8220;gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs&#8221;</a>.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus"><a class="anchor" href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus"></a><a class="link" href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">24.22.4.4. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></h5>
<div class="paragraph">
<p>It would be amazing to analyze a simple example with interconnect packets possibly invalidating caches of other CPUs.</p>
</div>
<div class="paragraph">
<p>To observe it we could create one well controlled workload with instructions that flush memory, and run it on two CPUs.</p>
</div>
<div class="paragraph">
<p>If we don&#8217;t use such instructions that flush memory, we would only see the interconnect at work when caches run out.</p>
</div>
<div class="paragraph">
<p>For this study, we will use the same CLI as <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a> but with multiple CPUs and a multithreaded which shares a variable across threads.</p>
</div>
<div class="paragraph">
<p>We can use <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic.c">userland/c/atomic.c</a> (see also <a href="#c-multithreading">C multithreading</a>) at LKMC 7c01b29f1ee7da878c7cc9cb4565f3f3cf516a92 and gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 was with as in <a href="#detailed-gem5-analysis-of-how-data-races-happen">Detailed gem5 analysis of how data races happen</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --cli-args '2 10' \
  --cpus 3 \
  --emulator gem5 \
  --trace FmtFlag,Cache,DRAM,ExecAll,XBar \
  --userland userland/c/atomic.c \
  -- \
  --caches \
;</pre>
</div>
</div>
<div class="paragraph">
<p>The <a href="#gem5-config-dot"><code>config.dot.svg</code></a> now looks like this but with 3 CPUs instead of 2:</p>
</div>
<div id="config-dot-svg-timingsimplecpu-caches-2-cpus" class="imageblock">
<div class="content">
<img src="https://raw.githubusercontent.com/cirosantilli/media/master/gem5_config_TimingSimpleCPU_caches_2_CPUs_12c917de54145d2d50260035ba7fa614e25317a3.svg?sanitize=true" alt="gem5 config TimingSimpleCPU caches 2 CPUs 12c917de54145d2d50260035ba7fa614e25317a3" height="600">
</div>
<div class="title">Figure 5. <code>config.dot.svg</code> for a system with two TimingSimpleCPU with caches.</div>
</div>
<div class="paragraph">
<p>Once again we focus on the shared function region <code>my_thread_main</code> which is where the interesting cross core memory collisions will be happening.</p>
</div>
<div class="paragraph">
<p>As a maybe-not-so-interesting, we have a look at the very first <code>my_thread_main</code> icache hit points:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>93946000: Cache: system.cpu1.icache: access for ReadReq [8b0:8b3] IF miss
93946000: Cache: system.cpu1.icache: createMissPacket: created ReadCleanReq [880:8bf] IF from ReadReq [8b0:8b3] IF
93946000: Cache: system.cpu1.icache: handleAtomicReqMiss: Sending an atomic ReadCleanReq [880:8bf] IF
93946000: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[5] packet ReadCleanReq [880:8bf] IF
93946000: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[5] packet ReadCleanReq [880:8bf] IF SF size: 1 lat: 1
93946000: Cache: system.cpu0.icache: handleSnoop: snoop hit for ReadCleanReq [880:8bf] IF, old state is state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x22 way: 0
93946000: Cache: system.cpu0.icache: new state is state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x22 way: 0
93946000: DRAM: system.mem_ctrls: recvAtomic: ReadCleanReq 0x880
93946000: Cache: system.cpu1.icache: handleAtomicReqMiss: Receive response: ReadResp [880:8bf] IF in state 0
93946000: Cache: system.cpu1.icache: Block addr 0x880 (ns) moving from state 0 to state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x22 way: 0
93946000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main    :   sub   sp, sp, #48        : IntAlu :  D=0x0000003fffd6b9a0  flags=(IsInteger)
93946500: Cache: system.cpu1.icache: access for ReadReq [8b4:8b7] IF hit state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x22 way: 0
93946500: Cache: system.cpu1.dcache: access for WriteReq [a19a8:a19af] hit state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0x14 set: 0x66 way: 0
93946500: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+4    :   str   x0, [sp, #8]       : MemWrite :  D=0x0000007ffffefc70 A=0x3fffd6b9a8  flags=(IsInteger|IsMemRef|IsStore)</pre>
</div>
</div>
<div class="paragraph">
<p>Now that we know how to read cache logs from <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a>, it is easier to understand what happened:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the physical address for <code>my_thread_main</code> is at 0x8b0, which gets requested is a miss, since it is the first time CPU1 goes near that region, since CPU1 was previously executing in standard library code far from our text segment</p>
</li>
<li>
<p>CPU0 already has has that cache line (0x880) in its cache at <a href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5">state E of MOESI</a>, so it snoops and moves to S. We can look up the logs to see exactly where CPU0 had previously read that address:</p>
<div class="literalblock">
<div class="content">
<pre>59135500: Cache: system.cpu0.icache: Block addr 0x880 (ns) moving from state 0 to state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x22 way: 0
59135500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[1] packet WritebackClean [8880:88bf]
59135500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[1] packet WritebackClean [8880:88bf] SF size: 0 lat: 1
59135500: DRAM: system.mem_ctrls: recvAtomic: WritebackClean 0x8880
59135500: ExecEnable: system.cpu0: A0 T0 : @frame_dummy    : stp</pre>
</div>
</div>
</li>
<li>
<p>the request does touch RAM, it does not get served by the other cache directly. CPU1 is now also at state S for the block</p>
</li>
<li>
<p>the second cache request from CPU1 is 4 bytes further ahead 0x8b4, and this time it is of course a hit.</p>
<div class="paragraph">
<p>Since this is an STR, it also does a dcache access, to 0xA19A8 in this case near its stack SP, and it is a hit, which is not surprising, since basically stack accesses are the very first thing any C code does, and there must be some setup code running on CPU1 before <code>my_thread_main</code>.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Now let&#8217;s look for the incremented integer address that is shared across threads. We know from <a href="#detailed-gem5-analysis-of-how-data-races-happen">Detailed gem5 analysis of how data races happen</a> that the read happens at <code>my_thread_main+36</code>, so searching for he first occurrence:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>93952500: Cache: system.cpu1.icache: access for ReadReq [8d4:8d7] IF hit state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x23 way: 0
93952500: Cache: system.cpu1.dcache: access for ReadReq [2060:2063] miss
93952500: Cache: system.cpu1.dcache: createMissPacket: created ReadSharedReq [2040:207f] from ReadReq [2060:2063]
93952500: Cache: system.cpu1.dcache: handleAtomicReqMiss: Sending an atomic ReadSharedReq [2040:207f]
93952500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[6] packet ReadSharedReq [2040:207f]
93952500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[6] packet ReadSharedReq [2040:207f] SF size: 0 lat: 1
93952500: DRAM: system.mem_ctrls: recvAtomic: ReadSharedReq 0x2040
93952500: Cache: system.cpu1.dcache: handleAtomicReqMiss: Receive response: ReadResp [2040:207f] in state 0
93952500: Cache: system.cpu1.dcache: Block addr 0x2040 (ns) moving from state 0 to state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0
93952500: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000000 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>so we determine its physical address of 0x2060. It was a miss, and then it went into E.</p>
</div>
<div class="paragraph">
<p>So we look ahead to the following accesses to that physical address, before CPU2 reaches that point of the code and starts making requests as well.</p>
</div>
<div class="paragraph">
<p>First there is the STR for the first LDR which is of course a hit:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>93954500: Cache: system.cpu1.dcache: access for WriteReq [2060:2063] hit state: 7 (E) valid: 1 writable: 1 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0
93954500: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+52    :   str   x1, [x0]           : MemWrite :  D=0x0000000000000001 A=0x411060  flags=(IsInteger|IsMemRef|IsStore)</pre>
</div>
</div>
<div class="paragraph">
<p>If found the line in E, so we presume that it moves it to M. Then the second read confirms that it was in M:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>93964500: Cache: system.cpu1.dcache: access for ReadReq [2060:2063] hit state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
93964500: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000001 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>and so on.</p>
</div>
<div class="paragraph">
<p>Now let&#8217;s jump to when CPU2 starts making requests.</p>
</div>
<div class="paragraph">
<p>The first time this happens is on its first LDR at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>94058500: Cache: system.cpu2.dcache: access for ReadReq [2060:2063] miss
94058500: Cache: system.cpu2.dcache: createMissPacket: created ReadSharedReq [2040:207f] from ReadReq [2060:2063]
94058500: Cache: system.cpu2.dcache: handleAtomicReqMiss: Sending an atomic ReadSharedReq [2040:207f]
94058500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[10] packet ReadSharedReq [2040:207f]
94058500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[10] packet ReadSharedReq [2040:207f] SF size: 1 lat: 1
94058500: Cache: system.cpu1.dcache: handleSnoop: snoop hit for ReadSharedReq [2040:207f], old state is state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
94058500: Cache: system.cpu1.dcache: new state is state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
94058500: CoherentXBar: system.membus: recvAtomicBackdoor: Not forwarding ReadSharedReq [2040:207f]
94058500: Cache: system.cpu2.dcache: handleAtomicReqMiss: Receive response: ReadResp [2040:207f] in state 0
94058500: Cache: system.cpu2.dcache: Block addr 0x2040 (ns) moving from state 0 to state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0
94058500: ExecEnable: system.cpu2: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>and from this we see:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPU1 moves from M to O</p>
</li>
<li>
<p>CPU2 moves from I to S</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It also appears that no DRAM was accessed since there are no logs for it, so did the XBar get the value directly from the other cache? TODO: why did the earlier <code>93946000: DRAM</code> read happened then, since CPU0 had the line when CPU1 asked for it?</p>
</div>
<div class="paragraph">
<p>The above log sequence also makes it clear that it is the XBar that maintains coherency: it appears that the CPU2 caches tells the XBar what it is doing, and then the XBar tells other caches on other CPUs about it, which leads CPU1 to move to O.</p>
</div>
<div class="paragraph">
<p>Then CPU1 hits its LDR on O:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>94060500: Cache: system.cpu1.dcache: access for ReadReq [2060:2063] hit state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
94060500: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>and then CPU2 writes moving to M and moving CPU1 to I:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>94060500: Cache: system.cpu2.dcache: access for WriteReq [2060:2063] hit state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0
94060500: Cache: system.cpu2.dcache: createMissPacket: created UpgradeReq [2040:207f] from WriteReq [2060:2063]
94060500: Cache: system.cpu2.dcache: handleAtomicReqMiss: Sending an atomic UpgradeReq [2040:207f]
94060500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[10] packet UpgradeReq [2040:207f]
94060500: CoherentXBar: system.membus: recvAtomicBackdoor: src system.membus.slave[10] packet UpgradeReq [2040:207f] SF size: 1 lat: 1
94060500: Cache: system.cpu1.dcache: handleSnoop: snoop hit for UpgradeReq [2040:207f], old state is state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
94060500: Cache: system.cpu1.dcache: new state is state: 0 (I) valid: 0 writable: 0 readable: 0 dirty: 0 | tag: 0xffffffffffffffff set: 0x81 way: 0
94060500: CoherentXBar: system.membus: recvAtomicBackdoor: Not forwarding UpgradeReq [2040:207f]
94060500: Cache: system.cpu2.dcache: handleAtomicReqMiss: Receive response: UpgradeResp [2040:207f] in state 5
94060500: Cache: system.cpu2.dcache: Block addr 0x2040 (ns) moving from state 5 to state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
94060500: ExecEnable: system.cpu2: A0 T0 : @my_thread_main+52    :   str   x1, [x0]           : MemWrite :  D=0x000000000000000a A=0x411060  flags=(IsInteger|IsMemRef|IsStore)</pre>
</div>
</div>
<div class="paragraph">
<p>and so on, they just keep fighting over that address and changing one another&#8217;s state.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus"><a class="anchor" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus"></a><a class="link" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">24.22.4.5. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a></h5>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a> but with <a href="#gem5-timingsimplecpu">gem5 <code>TimingSimpleCPU</code></a> and <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic/aarch64_add.c">userland/c/atomic/aarch64_add.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --arch aarch64 --optimization-level 3 --userland-build-id o3
./run \
  --arch aarch64 \
  --cli-args '2 1000' \
  --cpus 3 \
  --emulator gem5 \
  --trace FmtFlag,CacheAll,DRAM,Event,ExecAll,SimpleCPU,XBar \
  --userland userland/c/atomic/aarch64_add.c \
  --userland-build-id o3 \
  -- \
  --caches \
  --cpu-type TimingSimpleCPU \
;</pre>
</div>
</div>
<div class="paragraph">
<p>This is arguably the best experiment to study the <a href="#gem5-crossbar-interconnect">gem5 crossbar interconnect</a>.</p>
</div>
<div class="paragraph">
<p>We increase the loop count to 100 loops because 100 did not show memory conflicts. The output is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>expect 200
global 147</pre>
</div>
</div>
<div class="paragraph">
<p>Let&#8217;s double check what it compiles to with <a href="#disas">disas</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --userland userland/c/atomic/aarch64_add.c --userland-build-id o3 my_thread_main</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x0000000000400a70 &lt;+0&gt;:     03 00 40 f9     ldr     x3, [x0]
   0x0000000000400a74 &lt;+4&gt;:     63 01 00 b4     cbz     x3, 0x400aa0 &lt;my_thread_main+48&gt;
   0x0000000000400a78 &lt;+8&gt;:     82 00 00 d0     adrp    x2, 0x412000 &lt;malloc@got.plt&gt;
   0x0000000000400a7c &lt;+12&gt;:    42 a0 01 91     add     x2, x2, #0x68
   0x0000000000400a80 &lt;+16&gt;:    00 00 80 d2     mov     x0, #0x0                        // #0
   0x0000000000400a84 &lt;+20&gt;:    1f 20 03 d5     nop
   0x0000000000400a88 &lt;+24&gt;:    41 00 40 f9     ldr     x1, [x2]
   0x0000000000400a8c &lt;+28&gt;:    21 04 00 91     add     x1, x1, #0x1
   0x0000000000400a90 &lt;+32&gt;:    41 00 00 f9     str     x1, [x2]
   0x0000000000400a94 &lt;+36&gt;:    00 04 00 91     add     x0, x0, #0x1
   0x0000000000400a98 &lt;+40&gt;:    7f 00 00 eb     cmp     x3, x0
   0x0000000000400a9c &lt;+44&gt;:    68 ff ff 54     b.hi    0x400a88 &lt;my_thread_main+24&gt;  // b.pmore
   0x0000000000400aa0 &lt;+48&gt;:    00 00 80 52     mov     w0, #0x0                        // #0
   0x0000000000400aa4 &lt;+52&gt;:    c0 03 5f d6     ret</pre>
</div>
</div>
<div class="paragraph">
<p>Grepping the logs with <code>grep '@my_thread_main\+24</code> shows where the first non-atomic interleaves happen at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[many other CPU1 hits]
471199000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x000000000000002e A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)
471207000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x000000000000002f A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)
471202000: ExecEnable: system.cpu2: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x000000000000002f A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)
471239000: ExecEnable: system.cpu2: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x0000000000000030 A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)
471228000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x0000000000000030 A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)
471269000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x0000000000000031 A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>after a long string of cpu1 hits, since CPU1 was forked first and therefore had more time to run that operation.</p>
</div>
<div class="paragraph">
<p>From those and logs around we deduce that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the shared address of interest is 0x412068</p>
</li>
<li>
<p>the physical address is 0x2068</p>
</li>
<li>
<p>it fits into the cache line for 0x2040:0x207f</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>With that guide, we look at the fuller logs around that region of interest. With start at the first ifetch that CPU2 does for our LDR of interest at 0x400a88:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471201000: SimpleCPU: system.cpu2: Fetch
471201000: SimpleCPU: system.cpu2: Translating address 0x400a88</pre>
</div>
</div>
<div class="paragraph">
<p>Things get a bit interleaved with CPU1, but soon afterwards we see the CPU2 make its memory request to the cache:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471202000: Event: Event_134: Timing CPU icache tick 134 executed @ 471202000
471202000: SimpleCPU: system.cpu2: Complete ICache Fetch for addr 0xa88
471202000: Cache: system.cpu2.dcache: access for ReadReq [2068:206f] D=c879334bb1550000 num=266073 miss
471202000: CachePort: system.cpu2.dcache.mem_side: Scheduling send event at 471203000
471202000: Event: system.cpu2.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 140 scheduled @ 471203000</pre>
</div>
</div>
<div class="paragraph">
<p>Before the request moves on, some CPU1 action happens: a CPU1 is sending its data out! It hit the cache, and now we confirm that the cache is in <a href="#moesi">state: M</a> as expected, since CPU1 had already been previously writting repeatedly to that address:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471202000: Event: Event_87: Timing CPU icache tick 87 executed @ 471202000
471202000: SimpleCPU: system.cpu1: Complete ICache Fetch for addr 0xa90
471202000: Cache: system.cpu1.dcache: access for WriteReq [2068:206f] D=2f00000000000000 num=266074 hit state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
471202000: CacheVerbose: system.cpu1.dcache: satisfyRequest for WriteReq [2068:206f] D=2f00000000000000 num=266074 (write)
471202000: Event: system.cpu1.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 91 scheduled @ 471203000</pre>
</div>
</div>
<div class="paragraph">
<p>Immediately afterwards, CPU1 gets its reply from the cache, which is fast as that was a hit, and its STR finishes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471203000: Event: system.cpu1.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 91 executed @ 471203000
471203000: SimpleCPU: system.cpu1.dcache_port: Received load/store response 0x2068
471203000: Event: Event_89: Timing CPU dcache tick 89 scheduled @ 471203000
471203000: Event: Event_89: Timing CPU dcache tick 89 executed @ 471203000
471202000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+32    :   str   x1, [x2]           : MemWrite :  D=0x000000000000002f A=0x412068  flags=(IsInteger|IsMemRef|IsStore)</pre>
</div>
</div>
<div class="paragraph">
<p>Now we approach the crux of this example: cpu2 dcache decides to forward its read request:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471203000: Event: system.cpu2.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 140 executed @ 471203000
471203000: Cache: system.cpu2.dcache: sendMSHRQueuePacket: MSHR ReadReq [2068:206f] D=c879334bb1550000 num=266073
471203000: Cache: system.cpu2.dcache: createMissPacket: created ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076 from ReadReq [2068:206f] D=c879334bb1550000 num=266073</pre>
</div>
</div>
<div class="paragraph">
<p>Here, CPU2 dcache finally forwards to the XBar its request via the <a href="#gem5-mshr">gem5 <code>MSHR</code></a> mechanism as in <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a></p>
</div>
<div class="paragraph">
<p><code>createMissPacket</code> creates a new packet for the cache request with a different type: <code>ReadSharedReq</code> instead of the original <code>ReadReq</code>, and then it sends that packet into <a href="#gem5-crossbar-interconnect"><code>CoherentXBar</code></a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471203000: CoherentXBar: system.membus: recvTimingReq: src system.membus.slave[10] packet ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076
471203000: SnoopFilter: system.membus.snoop_filter: lookupRequest: src system.membus.slave[10] packet ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076
471203000: SnoopFilter: system.membus.snoop_filter: lookupRequest:   SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000
471203000: SnoopFilter: system.membus.snoop_filter: lookupRequest:   new SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000
471203000: CoherentXBar: system.membus: recvTimingReq: src system.membus.slave[10] packet ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076 SF size: 1 lat: 1
471203000: CoherentXBar: system.membus: forwardTiming for ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076</pre>
</div>
</div>
<div class="paragraph">
<p>The XBar receives the request, and notices that CPU1 cares about it, because obviously it has that line from previous writes, so the XBar forwards the exact same request to the CPU1 dcache:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471203000: CacheVerbose: system.cpu1.dcache: recvTimingSnoopReq: for ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076
471203000: CacheVerbose: system.cpu1.dcache: handleSnoop: for ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076
471203000: Cache: system.cpu1.dcache: handleSnoop: snoop hit for ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076, old state is state: f (M) valid: 1 writable: 1 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
471203000: Cache: system.cpu1.dcache: new state is state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
471203000: Cache: system.cpu1.dcache: doTimingSupplyResponse: for ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076
471203000: CacheVerbose: system.cpu1.dcache: doTimingSupplyResponse: created response: ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078 tick: 471212000
471203000: Event: system.cpu1.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 94 scheduled @ 471212000
471203000: CoherentXBar: system.membus: recvTimingReq: Not forwarding ReadSharedReq [2040:207f] D=40cfe14bb15500005b323036383a323036665d20443d63383739333334626231353530303030206e756d3d32363630373300000000016d6973730a0000000000 num=266076</pre>
</div>
</div>
<div class="paragraph">
<p>and from this we see that this read request from CPU2 made a cache from CPU1 go <a href="#moesi">from M to O</a>! <a href="#cache-coherence">Cache coherence</a> is being maintained!</p>
</div>
<div class="paragraph">
<p>Furthermore, it also suggests that now CPU1 is going to supply the response to CPU2 directly from its cache, and the memory request will be suppressed! As mentioned in lecture notes from <a href="#cache-coherence">Cache coherence</a>, we know that this is one of the ways that cache coherence may be maintained in MOESI-like protocols.</p>
</div>
<div class="paragraph">
<p>After this point, CPU1 continues to go around the loop. After a few instructions we don&#8217;t care about, we once again reach the LDR:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471207000: SimpleCPU: system.cpu1: Complete ICache Fetch for addr 0xa88
471207000: Cache: system.cpu1.dcache: access for ReadReq [2068:206f] D=c879334bb1550000 num=266082 hit state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
471207000: Event: system.cpu1.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 91 scheduled @ 471208000
471208000: Event: system.cpu1.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 91 executed @ 471208000
471208000: SimpleCPU: system.cpu1.dcache_port: Received load/store response 0x2068
471208000: Event: Event_89: Timing CPU dcache tick 89 scheduled @ 471208000
471208000: Event: Event_89: Timing CPU dcache tick 89 executed @ 471208000
471207000: ExecEnable: system.cpu1: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x000000000000002f A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>but it is immediately satisfied since the line is already in <code>O</code>, and nothing needs to be sent out to the bus since it&#8217;s a read.</p>
</div>
<div class="paragraph">
<p>Then the add 1 runs entirely from cache of course, and then CPU1 starts its STR:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471210000: Event: Event_87: Timing CPU icache tick 87 executed @ 471210000
471210000: SimpleCPU: system.cpu1: Complete ICache Fetch for addr 0xa90
471210000: Cache: system.cpu1.dcache: access for WriteReq [2068:206f] D=3000000000000000 num=266085 hit state: d (O) valid: 1 writable: 0 readable: 1 dirty: 1 | tag: 0 set: 0x81 way: 0
471210000: CachePort: system.cpu1.dcache.mem_side: Scheduling send event at 471211000
471210000: Event: system.cpu1.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 93 scheduled @ 471211000</pre>
</div>
</div>
<div class="paragraph">
<p>In parallel, the CPU1 snoop response to the CPU2 LDR that had been previously sent reaches the XBar:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471212000: Event: system.cpu1.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 94 executed @ 471212000
471212000: CoherentXBar: system.membus: recvTimingSnoopResp: src system.membus.slave[6] packet ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078
471212000: SnoopFilter: system.membus.snoop_filter: updateSnoopResponse: rsp system.membus.slave[6] req system.membus.slave[10] packet ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078
471212000: SnoopFilter: system.membus.snoop_filter: updateSnoopResponse:   old SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000
471212000: SnoopFilter: system.membus.snoop_filter: updateSnoopResponse:   new SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101000
471212000: CoherentXBar: system.membus: recvTimingSnoopResp: src system.membus.slave[6] packet ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078 FWD RESP
471212000: Event: system.membus.slave[10]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 186 scheduled @ 471214000
471212000: Event: system.membus.respLayer10.wrapped_function_event: EventFunctionWrapped 187 scheduled @ 471217000
471212000: BaseXBar: system.membus.respLayer10: The crossbar layer is now busy from tick 471212000 to 471217000</pre>
</div>
</div>
<div class="paragraph">
<p>We know that it is the same one based on the packet <code>num=</code> match.</p>
</div>
<div class="paragraph">
<p>And just after that, by coincidence, the CPU1 STR write request also starts going to the XBar:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471212001: Event: system.cpu1.dcache.mem_side-MemSidePort.wrapped_function_event: EventFunctionWrapped 93 executed @ 471212001
471212001: Cache: system.cpu1.dcache: sendMSHRQueuePacket: MSHR WriteReq [2068:206f] D=3000000000000000 num=266085
471212001: Cache: system.cpu1.dcache: createMissPacket: created UpgradeReq [2040:207f] D= num=266086 from WriteReq [2068:206f] D=3000000000000000 num=266085
471212001: CoherentXBar: system.membus: recvTimingReq: src system.membus.slave[6] packet UpgradeReq [2040:207f] D= num=266086
471212001: SnoopFilter: system.membus.snoop_filter: lookupRequest: src system.membus.slave[6] packet UpgradeReq [2040:207f] D= num=266086
471212001: SnoopFilter: system.membus.snoop_filter: lookupRequest:   SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101000
471212001: SnoopFilter: system.membus.snoop_filter: lookupRequest:   new SF value 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000101000
471212001: CoherentXBar: system.membus: recvTimingReq: src system.membus.slave[6] packet UpgradeReq [2040:207f] D= num=266086 SF size: 1 lat: 1
471212001: CoherentXBar: system.membus: forwardTiming for UpgradeReq [2040:207f] D= num=266086
471212001: CacheVerbose: system.cpu2.dcache: recvTimingSnoopReq: for UpgradeReq [2040:207f] D= num=266086
471212001: Cache: global: handleSnoop for UpgradeReq [2040:207f] D= num=266086
471212001: CacheVerbose: system.cpu2.dcache: handleSnoop: for UpgradeReq [2040:207f] D= num=266086
471212001: CacheVerbose: system.cpu2.dcache: handleSnoop: snoop miss for UpgradeReq [2040:207f] D= num=266086</pre>
</div>
</div>
<div class="paragraph">
<p>This time, we can see that the <code>WriteReq</code> gets turned into an <code>UpgradeReq</code> by the cache.</p>
</div>
<div class="paragraph">
<p>It does not however change the CPU2 cacheline state, because the CPU2 cache is not yet valid line because LDR reply still hasn&#8217;t come back! We see on the source code:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Cache::handleSnoop(PacketPtr pkt, CacheBlk *blk, bool is_timing,
                   bool is_deferred, bool pending_inval)
{
    } else if (!blk_valid) {
        DPRINTF(CacheVerbose, "%s: snoop miss for %s\n", __func__,
                pkt-&gt;print());</pre>
</div>
</div>
<div class="paragraph">
<p>At last, the CPU1 snoop reply reaches the CPU2 dcache with the (now old <code>2f</code>) data:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471214000: Event: system.membus.reqLayer0.wrapped_function_event: EventFunctionWrapped 164 executed @ 471214000
471214000: Event: system.membus.slave[10]-RespPacketQueue.wrapped_function_event: EventFunctionWrapped 186 executed @ 471214000
471214000: Cache: system.cpu2.dcache: recvTimingResp: Handling response ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078
471214000: Cache: system.cpu2.dcache: Block for addr 0x2040 being updated in Cache
471214000: CacheRepl: system.cpu2.dcache: Replacement victim: state: 0 (I) valid: 0 writable: 0 readable: 0 dirty: 0 | tag: 0xffffffffffffffff set: 0x81 way: 0
471214000: Cache: system.cpu2.dcache: Block addr 0x2040 (ns) moving from state 0 to state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0</pre>
</div>
</div>
<div class="paragraph">
<p>On the above, we see that this initially moves the cache to S state.</p>
</div>
<div class="paragraph">
<p>However, remember that after CPU2 started its LDR, CPU1 did an STR, and that STR was already snooped by CPU2 above? Well, the MSHR or the cache had noted that down, and now it proceeds to invalidate the line:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471214000: Cache: system.cpu2.dcache: serviceMSHRTargets: updated cmd to ReadRespWithInvalidate [2068:206f] D=2f00000000000000 num=266073
471214000: Event: system.cpu2.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 138 scheduled @ 471215000
471214000: Cache: system.cpu2.dcache: processing deferred snoop...
471214000: CacheVerbose: system.cpu2.dcache: handleSnoop: for UpgradeReq [2040:207f] D= num=266087
471214000: Cache: system.cpu2.dcache: handleSnoop: snoop hit for UpgradeReq [2040:207f] D= num=266087, old state is state: 5 (S) valid: 1 writable: 0 readable: 1 dirty: 0 | tag: 0 set: 0x81 way: 0
471214000: Cache: system.cpu2.dcache: new state is state: 0 (I) valid: 0 writable: 0 readable: 0 dirty: 0 | tag: 0xffffffffffffffff set: 0x81 way: 0
471214000: CacheVerbose: system.cpu2.dcache: recvTimingResp: Leaving with ReadResp [2040:207f] D=700640000000000070064000000000000000000000000000000000000000000000000000000000002f0000000000000000000000000000000000000000000000 num=266078</pre>
</div>
</div>
<div class="paragraph">
<p>It is a bit funny, but we see that at the same time, both the response arrived with the data, and the cache gets invalidated with a delay. The MSHR kept track of that for us. On the above logs, actually <code>Cache: global: handleSnoop</code> is the line in question.</p>
</div>
<div class="paragraph">
<p>And at long long last, the CPU2 LDR finishes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>471215000: Event: system.cpu2.dcache.cpu_side-CpuSidePort.wrapped_function_event: EventFunctionWrapped 138 executed @ 471215000
471215000: SimpleCPU: system.cpu2.dcache_port: Received load/store response 0x2068
471215000: Event: Event_136: Timing CPU dcache tick 136 scheduled @ 471215000
471215000: Event: Event_136: Timing CPU dcache tick 136 executed @ 471215000
471202000: ExecEnable: system.cpu2: A0 T0 : @my_thread_main+24    :   ldr   x1, [x2]           : MemRead :  D=0x000000000000002f A=0x412068  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>We note therefore that no DRAM access was involved, one cache services the other directly!</p>
</div>
<div class="paragraph">
<p>Tested on LKMC 4f82f79be7b0717c12924f4c9b7c4f46f8f18e2f + 1, gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 with a hack to add packet IDs and data to <code>Packet::print</code>.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus-and-ruby"><a class="anchor" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus-and-ruby"></a><a class="link" href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus-and-ruby">24.22.4.6. gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs and Ruby</a></h5>
<div class="paragraph">
<p>Now let&#8217;s do the exact same we did for <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs</a>, but with <a href="#gem5-ruby-build">Ruby</a> rather than the classic system and TimingSimpleCPU (atomic does not work with Ruby)</p>
</div>
<div class="paragraph">
<p>Since we have fully understood coherency in that previous example, it should now be easier to understand what is going on with Ruby:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --cli-args '2 10' \
  --cpus 3 \
  --emulator gem5 \
  --trace FmtFlag,DRAM,ExecAll,Ruby \
  --userland userland/c/atomic.c \
  -- \
  --cpu-type TimingSimpleCPU \
  --ruby \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Note that now the <code>--trace Cache,XBar</code> flags have no effect, since Ruby replaces those classic memory model components entirely with the Ruby version, so we enable the <code>Ruby</code> flag version instead. Note however that this flag is very verbose and produces about 10x more output than the classic memory experiment.</p>
</div>
<div class="paragraph">
<p>Also remember that ARM&#8217;s default Ruby protocol is <code>'MOESI_CMP_directory'</code>.</p>
</div>
<div class="paragraph">
<p>First we note that the output of the experiment is the same:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>atomic 20
non-atomic 19</pre>
</div>
</div>
<div class="paragraph">
<p>TODO</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis"><a class="anchor" href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis"></a><a class="link" href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis">24.22.4.7. gem5 event queue MinorCPU syscall emulation freestanding example analysis</a></h5>
<div class="paragraph">
<p>The events <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis">for the Atomic CPU</a> were pretty simple: basically just ticks.</p>
</div>
<div class="paragraph">
<p>But as we venture into <a href="#gem5-cpu-types">more complex CPU models</a> such as <code>MinorCPU</code>, the events get much more complex and interesting.</p>
</div>
<div class="paragraph">
<p>The memory system system part must be similar to that of <code>TimingSimpleCPU</code> that we previously studied <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis</a>: the main thing we want to see is how the CPU pipeline speeds up execution by preventing some memory stalls.</p>
</div>
<div class="paragraph">
<p>The <a href="#gem5-config-ini"><code>config.dot.svg</code></a> also indicates that: everything is exactly as in <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a>, except that the CPU is a <code>MinorCPU</code> instead of <code>TimingSimpleCPU</code>, and the <code>--caches</code> are now mandatory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace FmtFlag,Cache,Event,ExecAll,Minor \
  --trace-stdout \
  -- \
  --cpu-type MinorCPU \
  --caches \
;</pre>
</div>
</div>
<div class="paragraph">
<p>and here&#8217;s a handy link to the source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/hello.S">userland/arch/aarch64/freestanding/linux/hello.S</a>.</p>
</div>
<div class="paragraph">
<p>On LKMC ce3ea9faea95daf46dea80d4236a30a0891c3ca5 gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 we see the following.</p>
</div>
<div class="paragraph">
<p>First there is a missed instruction fetch for the initial entry address which we know from <a href="#gem5-event-queue-timingsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches">gem5 event queue TimingSimpleCPU syscall emulation freestanding example analysis with caches</a> is the virtual address 0x400078 which maps to physical 0x78:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    500: Cache: system.cpu.icache: access for ReadReq [40:7f] IF miss</pre>
</div>
</div>
<div class="paragraph">
<p>The memory request comes back later on at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  77000: Cache: system.cpu.icache: recvTimingResp: Handling response ReadResp [40:7f] IF</pre>
</div>
</div>
<div class="paragraph">
<p>and soon after the CPU also ifetches across the barrier:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  79000: Cache: system.cpu.icache: access for ReadReq [80:bf] IF miss</pre>
</div>
</div>
<div class="paragraph">
<p>TODO why? We have 0x78 and 0x7c, and those should be it since we <a href="#gem5-functional-units">are dual issue</a>, right? Is this prefetching at work?</p>
</div>
<div class="paragraph">
<p>Later on we see the first instruction, our <a href="#arm-mov-instruction">MOVZ</a>, was decoded:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  80000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/1/1.1 pc: 0x400078 (movz) to FU: 0</pre>
</div>
</div>
<div class="paragraph">
<p>and that issue succeeds, because the functional unit 0 (FU 0) is an <code>IntAlu</code> as shown at <a href="#gem5-functional-units">gem5 functional units</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  80000: MinorExecute: system.cpu.execute: Issuing inst: 0/1.1/1/1.1 pc: 0x400078 (movz) into FU 0</pre>
</div>
</div>
<div class="paragraph">
<p>At the very same tick, the second instruction is also decoded, our <a href="#arm-adr-instruction">ADR</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  80000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/1/2.2 pc: 0x40007c (adr) to FU: 0
  80000: MinorExecute: system.cpu.execute: Can't issue as FU: 0 is already busy
  80000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/1/2.2 pc: 0x40007c (adr) to FU: 1
  80000: MinorExecute: system.cpu.execute: Issuing inst: 0/1.1/1/2.2 pc: 0x40007c (adr) into FU 1</pre>
</div>
</div>
<div class="paragraph">
<p>This is also an <code>IntAlu</code> instruction, and it can&#8217;t run on FU 0 because the first instruction is already running there. But to our luck, FU 1 is also an <code>IntAlu</code> unit, and so it runs there.</p>
</div>
<div class="paragraph">
<p>Crap, those Minor logs should say what <code>OpClass</code> each instruction is, that would make things clearer.</p>
</div>
<div class="paragraph">
<p>TODO what is that <code>0/1.1/1/1.1</code> notation that shows up everywhere? Must be important, let&#8217;s look at the source.</p>
</div>
<div class="paragraph">
<p>Soon after (3 ticks later, so guessing due to <code>opLat=3</code>?), the execution appears to be over already since we see the <code>ExecAll</code> come through, which generally happens at the very end:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  81500: MinorExecute: system.cpu.execute: Attempting to commit [tid:0]
  81500: MinorExecute: system.cpu.execute: Committing micro-ops for interrupt[tid:0]
  81500: MinorExecute: system.cpu.execute: Trying to commit canCommitInsts: 1
  81500: MinorExecute: system.cpu.execute: Trying to commit from FUs
  81500: MinorExecute: global: ExecContext setting PC: (0x400078=&gt;0x40007c).(0=&gt;1)
  81500: MinorExecute: system.cpu.execute: Committing inst: 0/1.1/1/1.1 pc: 0x400078 (movz)
  81500: MinorExecute: system.cpu.execute: Unstalling 0 for inst 0/1.1/1/1.1
  81500: MinorExecute: system.cpu.execute: Completed inst: 0/1.1/1/1.1 pc: 0x400078 (movz)
  81500: MinorScoreboard: system.cpu.execute.scoreboard0: Clearing inst: 0/1.1/1/1.1 pc: 0x400078 (movz) regIndex: 0 final numResults: 0
  81500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #1, #0        : IntAlu :  D=0x0000000000000001  FetchSeq=1  CPSeq=1  flags=(IsInteger)
  81500: MinorExecute: system.cpu.execute: Trying to commit canCommitInsts: 1
  81500: MinorExecute: system.cpu.execute: Trying to commit from FUs
  81500: MinorExecute: global: ExecContext setting PC: (0x40007c=&gt;0x400080).(0=&gt;1)
  81500: MinorExecute: system.cpu.execute: Committing inst: 0/1.1/1/2.2 pc: 0x40007c (adr)
  81500: MinorExecute: system.cpu.execute: Unstalling 1 for inst 0/1.1/1/2.2
  81500: MinorExecute: system.cpu.execute: Completed inst: 0/1.1/1/2.2 pc: 0x40007c (adr)
  81500: MinorScoreboard: system.cpu.execute.scoreboard0: Clearing inst: 0/1.1/1/2.2 pc: 0x40007c (adr) regIndex: 1 final numResults: 0
  81500: MinorExecute: system.cpu.execute: Reached inst commit limit
  81500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   adr   x1, #28            : IntAlu :  D=0x0000000000400098  FetchSeq=2  CPSeq=2  flags=(IsInteger)</pre>
</div>
</div>
<div class="paragraph">
<p>The ifetch for the third instruction returns at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 129000: Cache: system.cpu.icache: recvTimingResp: Handling response ReadResp [80:bf] IF</pre>
</div>
</div>
<div class="paragraph">
<p>so now we are ready to run the third and fourth instructions of the program:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    ldr x2, =len
    mov x8, 64</pre>
</div>
</div>
<div class="paragraph">
<p>The <a href="#arm-ldr-instruction">LDR</a> goes all the way down to FU 6 which is the memory one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 0
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 0 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 1
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 1 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 2
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 2 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 3
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 3 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 4
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 4 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 5
 132000: MinorExecute: system.cpu.execute: Can't issue as FU: 5 isn't capable
 132000: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) to FU: 6
 132000: MinorExecute: system.cpu.execute: Issuing inst: 0/1.1/2/3.3 pc: 0x400080 (ldr) into FU 6</pre>
</div>
</div>
<div class="paragraph">
<p>and then the MOV issue follows soon afterwards (TODO why not at the same time like for the previous pair?):</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 132500: MinorExecute: system.cpu.execute: Trying to issue inst: 0/1.1/2/4.4 pc: 0x400084 (movz) to FU: 0
 132500: MinorExecute: system.cpu.execute: Issuing inst: 0/1.1/2/4.4 pc: 0x400084 (movz) into FU 0</pre>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis-hazard"><a class="anchor" href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis-hazard"></a><a class="link" href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis-hazard">24.22.4.7.1. gem5 event queue MinorCPU syscall emulation freestanding example analysis: hazard</a></h6>
<div class="paragraph">
<p>TODO like <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard</a> but with the hazard.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis">24.22.4.8. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis</a></h5>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis">gem5 event queue MinorCPU syscall emulation freestanding example analysis</a> but even more complex since for the <a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a>!</p>
</div>
<div class="paragraph">
<p>The key new <a href="#gem5-tracing">debug flag</a> is <code>O3CPUAll</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace FmtFlag,Cache,Event,ExecAll,O3CPUAll \
  --trace-stdout \
  -- \
  --cpu-type DerivO3CPU \
  --caches \
;</pre>
</div>
</div>
<div class="paragraph">
<p>The output is huge and contains about 7 thousand lines!!!</p>
</div>
<div class="paragraph">
<p>This section and children are tested at LKMC 144a552cf926ea630ef9eadbb22b79fe2468c456.</p>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless">24.22.4.8.1. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless</a></h6>
<div class="paragraph">
<p>Let&#8217;s  have a look at the arguably simplest example <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/hazardless.S">userland/arch/aarch64/freestanding/linux/hazardless.S</a>.</p>
</div>
<div class="paragraph">
<p>First let&#8217;s start with a <a href="#gem5-util-o3-pipeview-py-o3-pipeline-viewer">gem5 util/o3-pipeview.py O3 pipeline viewer</a> visualization:</p>
</div>
<div id="hazardless-o3-pipeline" class="listingblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x00400078.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x0040007c.0 movz x1, #1, #0           [         2]
[....................fdn.ic.r....................................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [         3]
[....................fdn.ic.r....................................................]-(         120000) 0x00400084.0 movz x3, #3, #0           [         4]
[....................fdn.ic.r....................................................]-(         120000) 0x00400088.0 movz x4, #4, #0           [         5]
[....................fdn.ic.r....................................................]-(         120000) 0x0040008c.0 movz x5, #5, #0           [         6]
[....................fdn.ic.r....................................................]-(         120000) 0x00400090.0 movz x6, #6, #0           [         7]
[....................fdn.ic.r....................................................]-(         120000) 0x00400094.0 movz x7, #7, #0           [         8]
[....................fdn.pic.r...................................................]-(         120000) 0x00400098.0 movz x8, #8, #0           [         9]
[....................fdn.pic.r...................................................]-(         120000) 0x0040009c.0 movz x9, #9, #0           [        10]
[.....................fdn.ic.r...................................................]-(         120000) 0x004000a0.0 movz x10, #10, #0         [        11]
[.....................fdn.ic.r...................................................]-(         120000) 0x004000a4.0 movz x11, #11, #0         [        12]
[.....................fdn.ic.r...................................................]-(         120000) 0x004000a8.0 movz x12, #12, #0         [        13]
[.....................fdn.ic.r...................................................]-(         120000) 0x004000ac.0 movz x13, #13, #0         [        14]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b0.0 movz x14, #14, #0         [        15]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b4.0 movz x15, #15, #0         [        16]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b8.0 movz x16, #16, #0         [        17]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000bc.0 movz x17, #17, #0         [        18]
[............................................fdn.ic.r............................]-(         160000) 0x004000c0.0 movz x18, #18, #0         [        19]
[............................................fdn.ic.r............................]-(         160000) 0x004000c4.0 movz x19, #19, #0         [        20]
[............................................fdn.ic.r............................]-(         160000) 0x004000c8.0 movz x20, #20, #0         [        21]
[............................................fdn.ic.r............................]-(         160000) 0x004000cc.0 movz x21, #21, #0         [        22]
[............................................fdn.ic.r............................]-(         160000) 0x004000d0.0 movz x22, #22, #0         [        23]
[............................................fdn.ic.r............................]-(         160000) 0x004000d4.0 movz x23, #23, #0         [        24]
[............................................fdn.pic.r...........................]-(         160000) 0x004000d8.0 movz x24, #24, #0         [        25]
[............................................fdn.pic.r...........................]-(         160000) 0x004000dc.0 movz x25, #25, #0         [        26]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e0.0 movz x26, #26, #0         [        27]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e4.0 movz x27, #27, #0         [        28]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e8.0 movz x28, #28, #0         [        29]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000ec.0 movz x29, #29, #0         [        30]
[.............................................fdn.pic.r..........................]-(         160000) 0x004000f0.0 movz x0, #0, #0           [        31]
[.............................................fdn.pic.r..........................]-(         160000) 0x004000f4.0 movz x1, #1, #0           [        32]
[.............................................fdn.pic.r..........................]-(         160000) 0x004000f8.0 movz x2, #2, #0           [        33]
[.............................................fdn.pic.r..........................]-(         160000) 0x004000fc.0 movz x3, #3, #0           [        34]</pre>
</div>
</div>
<div class="paragraph">
<p>The first of instructions has only two instructions because the first instruction is at address 0x400078, so only two instructions fit on that cache line, as the next cache line starts at 0x400080!</p>
</div>
<div class="paragraph">
<p>The initial <code>fdn</code> on top middle is likely bugged out, did it wrap around? But the rest makes sense.</p>
</div>
<div class="paragraph">
<p>From this, we clearly see that up to 8 instructions can be issued concurrently, which matches the default width values we had seen at <a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a>.</p>
</div>
<div class="paragraph">
<p>For example, we can clearly see how:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>movz x2</code> through to <code>movz x9</code> start running at the exact same time. TODO why does <code>mov x7</code> do <code>fdn.ic.r</code> while <code>mov x8</code> do <code>fdn.ic.r</code>? How are they different?</p>
</li>
<li>
<p><code>movz x10</code> through <code>movz x17</code> then starts running one step later. This second chunk is fully pipelined with the first instruction pack</p>
</li>
<li>
<p>then comes a pause while the next fetch comes back. This group of 16 instructions took up the entire 64-byte cacheline that had been read</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>First we can have a look at <code>ExecEnable</code> to get an initial ideal of how many instructions are run at one time:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  78500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  FetchSeq=1  CPSeq=1  flags=(IsInteger)
  78500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+4    :   movz   x1, #1, #0        : IntAlu :  D=0x0000000000000001  FetchSeq=2  CPSeq=2  flags=(IsInteger)

 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   movz   x2, #2, #0        : IntAlu :  D=0x0000000000000002  FetchSeq=3  CPSeq=3  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x3, #3, #0        : IntAlu :  D=0x0000000000000003  FetchSeq=4  CPSeq=4  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   movz   x4, #4, #0        : IntAlu :  D=0x0000000000000004  FetchSeq=5  CPSeq=5  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x5, #5, #0        : IntAlu :  D=0x0000000000000005  FetchSeq=6  CPSeq=6  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x6, #6, #0        : IntAlu :  D=0x0000000000000006  FetchSeq=7  CPSeq=7  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   movz   x7, #7, #0        : IntAlu :  D=0x0000000000000007  FetchSeq=8  CPSeq=8  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+32    :   movz   x8, #8, #0        : IntAlu :  D=0x0000000000000008  FetchSeq=9  CPSeq=9  flags=(IsInteger)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+36    :   movz   x9, #9, #0        : IntAlu :  D=0x0000000000000009  FetchSeq=10  CPSeq=10  flags=(IsInteger)

 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+40    :   movz   x10, #10, #0      : IntAlu :  D=0x000000000000000a  FetchSeq=11  CPSeq=11  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+44    :   movz   x11, #11, #0      : IntAlu :  D=0x000000000000000b  FetchSeq=12  CPSeq=12  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+48    :   movz   x12, #12, #0      : IntAlu :  D=0x000000000000000c  FetchSeq=13  CPSeq=13  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+52    :   movz   x13, #13, #0      : IntAlu :  D=0x000000000000000d  FetchSeq=14  CPSeq=14  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+56    :   movz   x14, #14, #0      : IntAlu :  D=0x000000000000000e  FetchSeq=15  CPSeq=15  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+60    :   movz   x15, #15, #0      : IntAlu :  D=0x000000000000000f  FetchSeq=16  CPSeq=16  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+64    :   movz   x16, #16, #0      : IntAlu :  D=0x0000000000000010  FetchSeq=17  CPSeq=17  flags=(IsInteger)
 130500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+68    :   movz   x17, #17, #0      : IntAlu :  D=0x0000000000000011  FetchSeq=18  CPSeq=18  flags=(IsInteger)</pre>
</div>
</div>
<div class="paragraph">
<p>This suggests 8, but remember that <code>ExecEnable</code> shows issue time labels, which do not coincide necessarily with commit times. As we saw in the pipeline viewer above, instructions 9 and 10 have one extra stage.</p>
</div>
<div class="paragraph">
<p>After the initial two execs from the first cache line, the full commit log chunk around the first group of six `ExecEnable`s looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 133500: Commit: system.cpu.commit: Getting instructions from Rename stage.
 133500: Commit: system.cpu.commit: Trying to commit instructions in the ROB.

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:3]
 133500: Commit: system.cpu.commit: [tid:0] [sn:3] Committing instruction with PC (0x400080=&gt;0x400084).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+8    :   movz   x2, #2, #0        : IntAlu :  D=0x0000000000000002  FetchSeq=3  CPSeq=3  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x400080=&gt;0x400084).(0=&gt;1), [sn:3]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400080=&gt;0x400084).(0=&gt;1) [sn:3]

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:4]
 133500: Commit: system.cpu.commit: [tid:0] [sn:4] Committing instruction with PC (0x400084=&gt;0x400088).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+12    :   movz   x3, #3, #0        : IntAlu :  D=0x0000000000000003  FetchSeq=4  CPSeq=4  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x400084=&gt;0x400088).(0=&gt;1), [sn:4]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400084=&gt;0x400088).(0=&gt;1) [sn:4]

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:5]
 133500: Commit: system.cpu.commit: [tid:0] [sn:5] Committing instruction with PC (0x400088=&gt;0x40008c).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+16    :   movz   x4, #4, #0        : IntAlu :  D=0x0000000000000004  FetchSeq=5  CPSeq=5  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x400088=&gt;0x40008c).(0=&gt;1), [sn:5]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400088=&gt;0x40008c).(0=&gt;1) [sn:5]

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:6]
 133500: Commit: system.cpu.commit: [tid:0] [sn:6] Committing instruction with PC (0x40008c=&gt;0x400090).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+20    :   movz   x5, #5, #0        : IntAlu :  D=0x0000000000000005  FetchSeq=6  CPSeq=6  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x40008c=&gt;0x400090).(0=&gt;1), [sn:6]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x40008c=&gt;0x400090).(0=&gt;1) [sn:6]

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:7]
 133500: Commit: system.cpu.commit: [tid:0] [sn:7] Committing instruction with PC (0x400090=&gt;0x400094).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+24    :   movz   x6, #6, #0        : IntAlu :  D=0x0000000000000006  FetchSeq=7  CPSeq=7  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x400090=&gt;0x400094).(0=&gt;1), [sn:7]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400090=&gt;0x400094).(0=&gt;1) [sn:7]

 133500: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:8]
 133500: Commit: system.cpu.commit: [tid:0] [sn:8] Committing instruction with PC (0x400094=&gt;0x400098).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+28    :   movz   x7, #7, #0        : IntAlu :  D=0x0000000000000007  FetchSeq=8  CPSeq=8  flags=(IsInteger)
 133500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x400094=&gt;0x400098).(0=&gt;1), [sn:8]
 133500: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x400094=&gt;0x400098).(0=&gt;1) [sn:8]

 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x400098=&gt;0x40009c).(0=&gt;1), [sn:9] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x40009c=&gt;0x4000a0).(0=&gt;1), [sn:10] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x4000a0=&gt;0x4000a4).(0=&gt;1), [sn:11] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x4000a4=&gt;0x4000a8).(0=&gt;1), [sn:12] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x4000a8=&gt;0x4000ac).(0=&gt;1), [sn:13] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Marking PC (0x4000ac=&gt;0x4000b0).(0=&gt;1), [sn:14] ready within ROB.
 133500: Commit: system.cpu.commit: [tid:0] Instruction [sn:9] PC (0x400098=&gt;0x40009c).(0=&gt;1) is head of ROB and ready to commit
 133500: Commit: system.cpu.commit: [tid:0] ROB has 10 insts &amp; 182 free entries.</pre>
</div>
</div>
<div class="paragraph">
<p><code>ROB</code> stands for <a href="#re-order-buffer">Re-order buffer</a>.</p>
</div>
<div class="paragraph">
<p><code>0x400080&#8658;0x400084</code> is an old/new PC address of the first committed instruction.</p>
</div>
<div class="paragraph">
<p>Another thing we can do, it to try to follow one of the instructions back as it goes through the pipeline. Searching for example for the address <code>0x400080</code>, we find:</p>
</div>
<div class="paragraph">
<p>The first mention of the address happens when is the fetch of the two initial instructions completes. TODO not sure why it doesn&#8217;t just also fetch the next cache line at the same time:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FullO3CPU: Ticking main, FullO3CPU.
  78500: Fetch: system.cpu.fetch: Running stage.
  78500: Fetch: system.cpu.fetch: Attempting to fetch from [tid:0]
  78500: Fetch: system.cpu.fetch: [tid:0] Icache miss is complete.
  78500: Fetch: system.cpu.fetch: [tid:0] Adding instructions to queue to decode.
  78500: DynInst: global: DynInst: [sn:1] Instruction created. Instcount for system.cpu = 1
  78500: Fetch: system.cpu.fetch: [tid:0] Instruction PC 0x400078 (0) created [sn:1].
  78500: Fetch: system.cpu.fetch: [tid:0] Instruction is:   movz   x0, #0, #0
  78500: Fetch: system.cpu.fetch: [tid:0] Fetch queue entry created (1/32).
  78500: DynInst: global: DynInst: [sn:2] Instruction created. Instcount for system.cpu = 2
  78500: Fetch: system.cpu.fetch: [tid:0] Instruction PC 0x40007c (0) created [sn:2].
  78500: Fetch: system.cpu.fetch: [tid:0] Instruction is:   movz   x1, #1, #0
  78500: Fetch: system.cpu.fetch: [tid:0] Fetch queue entry created (2/32).
  78500: Fetch: system.cpu.fetch: [tid:0] Issuing a pipelined I-cache access, starting at PC (0x400080=&gt;0x400084).(0=&gt;1).
  78500: Fetch: system.cpu.fetch: [tid:0] Fetching cache line 0x400080 for addr 0x400080</pre>
</div>
</div>
<div class="paragraph">
<p>so we observe that the first two instructions arrived, and the CPU noticed that 0x400080 hasn&#8217;t been fetched yet.</p>
</div>
<div class="paragraph">
<p>Then for several cycles that follow, the fetch stage just says that it is blocked on data returning:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FullO3CPU: Ticking main, FullO3CPU.
  79000: Fetch: system.cpu.fetch: Running stage.
  79000: Fetch: system.cpu.fetch: There are no more threads available to fetch from.
  79000: Fetch: system.cpu.fetch: [tid:0] Fetch is waiting cache response!</pre>
</div>
</div>
<div class="paragraph">
<p>At the same time, the execution of the initial 2 instructions progresses through the pipeline.</p>
</div>
<div class="paragraph">
<p>These progress up until:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  88000: O3CPU: system.cpu: Idle!</pre>
</div>
</div>
<div class="paragraph">
<p>at which point there are no more events scheduled besides waiting for the second cache line to come back.</p>
</div>
<div class="paragraph">
<p>After this, some time passes without events, and the next tick happens when the fetch data returns:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FullO3CPU: Ticking main, FullO3CPU.
 130000: Fetch: system.cpu.fetch: Running stage.
 130000: Fetch: system.cpu.fetch: Attempting to fetch from [tid:0]
 130000: Fetch: system.cpu.fetch: [tid:0] Icache miss is complete.
 130000: Fetch: system.cpu.fetch: [tid:0] Adding instructions to queue to decode.
 130000: DynInst: global: DynInst: [sn:3] Instruction created. Instcount for system.cpu = 1
 130000: Fetch: system.cpu.fetch: [tid:0] Instruction PC 0x400080 (0) created [sn:3].
 130000: Fetch: system.cpu.fetch: [tid:0] Instruction is:   movz   x2, #2, #0
 130000: Fetch: system.cpu.fetch: [tid:0] Fetch queue entry created (1/32).
 130000: DynInst: global: DynInst: [sn:4] Instruction created. Instcount for system.cpu = 2
 130000: Fetch: system.cpu.fetch: [tid:0] Instruction PC 0x400084 (0) created [sn:4].
 130000: Fetch: system.cpu.fetch: [tid:0] Instruction is:   movz   x3, #3, #0
 130000: Fetch: system.cpu.fetch: [tid:0] Fetch queue entry created (2/32).
 130000: DynInst: global: DynInst: [sn:5] Instruction created. Instcount for system.cpu = 3</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard">24.22.4.8.2. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard</a></h6>
<div class="paragraph">
<p>Now let&#8217;s do the same as in <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazardless">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazardless</a> but with a hazard: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/hazard.S">userland/arch/aarch64/freestanding/linux/hazard.S</a>.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x00400078.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x0040007c.0 movz x1, #1, #0           [         2]
[....................fdn.ic.r....................................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [         3]
[....................fdn.pic.r...................................................]-(         120000) 0x00400084.0 add x3, x2, #1            [         4]
[....................fdn.ic..r...................................................]-(         120000) 0x00400088.0 movz x4, #4, #0           [         5]
[....................fdn.ic..r...................................................]-(         120000) 0x0040008c.0 movz x5, #5, #0           [         6]
[....................fdn.ic..r...................................................]-(         120000) 0x00400090.0 movz x6, #6, #0           [         7]
[....................fdn.ic..r...................................................]-(         120000) 0x00400094.0 movz x7, #7, #0           [         8]
[....................fdn.ic..r...................................................]-(         120000) 0x00400098.0 movz x8, #8, #0           [         9]
[....................fdn.pic.r...................................................]-(         120000) 0x0040009c.0 movz x9, #9, #0           [        10]
[.....................fdn.ic.r...................................................]-(         120000) 0x004000a0.0 movz x10, #10, #0         [        11]
[.....................fdn.ic..r..................................................]-(         120000) 0x004000a4.0 movz x11, #11, #0         [        12]
[.....................fdn.ic..r..................................................]-(         120000) 0x004000a8.0 movz x12, #12, #0         [        13]
[.....................fdn.ic..r..................................................]-(         120000) 0x004000ac.0 movz x13, #13, #0         [        14]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b0.0 movz x14, #14, #0         [        15]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b4.0 movz x15, #15, #0         [        16]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000b8.0 movz x16, #16, #0         [        17]
[.....................fdn.pic.r..................................................]-(         120000) 0x004000bc.0 movz x17, #17, #0         [        18]
[............................................fdn.ic.r............................]-(         160000) 0x004000c0.0 movz x18, #18, #0         [        19]
[............................................fdn.ic.r............................]-(         160000) 0x004000c4.0 movz x19, #19, #0         [        20]
[............................................fdn.ic.r............................]-(         160000) 0x004000c8.0 movz x20, #20, #0         [        21]
[............................................fdn.ic.r............................]-(         160000) 0x004000cc.0 movz x21, #21, #0         [        22]
[............................................fdn.ic.r............................]-(         160000) 0x004000d0.0 movz x22, #22, #0         [        23]
[............................................fdn.ic.r............................]-(         160000) 0x004000d4.0 movz x23, #23, #0         [        24]
[............................................fdn.pic.r...........................]-(         160000) 0x004000d8.0 movz x24, #24, #0         [        25]
[............................................fdn.pic.r...........................]-(         160000) 0x004000dc.0 movz x25, #25, #0         [        26]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e0.0 movz x0, #0, #0           [        27]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e4.0 movz x8, #93, #0          [        28]</pre>
</div>
</div>
<div class="paragraph">
<p>TODO understand how the hazard happens in detail.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard4"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard4"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard4">24.22.4.8.3. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard4</a></h6>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard</a> but a hazard of depth 4: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/hazard.S">userland/arch/aarch64/freestanding/linux/hazard.S</a>.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x00400078.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x0040007c.0 movz x1, #1, #0           [         2]
[....................fdn.ic.r....................................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [         3]
[....................fdn.pic.r...................................................]-(         120000) 0x00400084.0 add x3, x2, #1            [         4]
[....................fdn.p.ic.r..................................................]-(         120000) 0x00400088.0 add x4, x3, #1            [         5]
[....................fdn.p..ic.r.................................................]-(         120000) 0x0040008c.0 add x5, x4, #1            [         6]
[....................fdn.p...ic.r................................................]-(         120000) 0x00400090.0 add x6, x5, #1            [         7]
[....................fdn.ic.....r................................................]-(         120000) 0x00400094.0 movz x7, #7, #0           [         8]
[....................fdn.ic.....r................................................]-(         120000) 0x00400098.0 movz x8, #8, #0           [         9]
[....................fdn.ic.....r................................................]-(         120000) 0x0040009c.0 movz x9, #9, #0           [        10]
[.....................fdn.ic....r................................................]-(         120000) 0x004000a0.0 movz x10, #10, #0         [        11]
[.....................fdn.ic....r................................................]-(         120000) 0x004000a4.0 movz x11, #11, #0         [        12]
[.....................fdn.ic....r................................................]-(         120000) 0x004000a8.0 movz x12, #12, #0         [        13]
[.....................fdn.ic....r................................................]-(         120000) 0x004000ac.0 movz x13, #13, #0         [        14]
[.....................fdn.ic.....r...............................................]-(         120000) 0x004000b0.0 movz x14, #14, #0         [        15]
[.....................fdn.pic....r...............................................]-(         120000) 0x004000b4.0 movz x15, #15, #0         [        16]
[.....................fdn.pic....r...............................................]-(         120000) 0x004000b8.0 movz x16, #16, #0         [        17]
[.....................fdn.pic....r...............................................]-(         120000) 0x004000bc.0 movz x17, #17, #0         [        18]
[............................................fdn.ic.r............................]-(         160000) 0x004000c0.0 movz x18, #18, #0         [        19]
[............................................fdn.ic.r............................]-(         160000) 0x004000c4.0 movz x19, #19, #0         [        20]
[............................................fdn.ic.r............................]-(         160000) 0x004000c8.0 movz x20, #20, #0         [        21]
[............................................fdn.ic.r............................]-(         160000) 0x004000cc.0 movz x21, #21, #0         [        22]
[............................................fdn.ic.r............................]-(         160000) 0x004000d0.0 movz x22, #22, #0         [        23]
[............................................fdn.ic.r............................]-(         160000) 0x004000d4.0 movz x23, #23, #0         [        24]
[............................................fdn.pic.r...........................]-(         160000) 0x004000d8.0 movz x24, #24, #0         [        25]
[............................................fdn.pic.r...........................]-(         160000) 0x004000dc.0 movz x25, #25, #0         [        26]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e0.0 movz x0, #0, #0           [        27]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000e4.0 movz x8, #93, #0          [        28]</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall">24.22.4.8.4. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall</a></h6>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-hazard">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: hazard</a> but now with an LDR stall: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/stall.S">userland/arch/aarch64/freestanding/linux/stall.S</a>.</p>
</div>
<div class="paragraph">
<p>We can see here that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the addition of a data section entry changed our previous address setup a bit, the entry point was now 0x004000b0 which fits 4 instructions in the cacheline instead of 2</p>
</li>
<li>
<p>the <a href="#arm-ldr-instruction">LDR</a> happens to be the fourth instruction, so it takes a long time to retire. The time is about 40k ticks, which is about the same time it takes for the instruction fetch as expected.</p>
</li>
<li>
<p>fetch does not continue past the LDR, and so nothing is gained in this particular example, since the next instructions haven&#8217;t been fetched from memory yet!</p>
</li>
</ul>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x004000b0.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x004000b4.0 movz x1, #1, #0           [         2]
[.ic.r........................................................................fdn]-(          40000) 0x004000b8.0 adr x2, #65780            [         3]
[.............................................................................fdn]-(          40000) 0x004000bc.0 ldr x3, [x2]              [         4]
[.pic............................................................................]-(          80000)     ...
[................................r...............................................]-(         120000)     ...
[....................fdn.ic......r...............................................]-(         120000) 0x004000c0.0 movz x4, #4, #0           [         5]
[....................fdn.ic......r...............................................]-(         120000) 0x004000c4.0 movz x5, #5, #0           [         6]
[....................fdn.ic......r...............................................]-(         120000) 0x004000c8.0 movz x6, #6, #0           [         7]
[....................fdn.ic......r...............................................]-(         120000) 0x004000cc.0 movz x7, #7, #0           [         8]
[....................fdn.ic......r...............................................]-(         120000) 0x004000d0.0 movz x8, #8, #0           [         9]
[....................fdn.ic......r...............................................]-(         120000) 0x004000d4.0 movz x9, #9, #0           [        10]
[....................fdn.pic.....r...............................................]-(         120000) 0x004000d8.0 movz x10, #10, #0         [        11]
[....................fdn.pic......r..............................................]-(         120000) 0x004000dc.0 movz x11, #11, #0         [        12]
[.....................fdn.ic......r..............................................]-(         120000) 0x004000e0.0 movz x12, #12, #0         [        13]
[.....................fdn.ic......r..............................................]-(         120000) 0x004000e4.0 movz x13, #13, #0         [        14]
[.....................fdn.ic......r..............................................]-(         120000) 0x004000e8.0 movz x14, #14, #0         [        15]
[.....................fdn.ic......r..............................................]-(         120000) 0x004000ec.0 movz x15, #15, #0         [        16]
[.....................fdn.pic.....r..............................................]-(         120000) 0x004000f0.0 movz x16, #16, #0         [        17]
[.....................fdn.pic.....r..............................................]-(         120000) 0x004000f4.0 movz x17, #17, #0         [        18]
[.....................fdn.pic.....r..............................................]-(         120000) 0x004000f8.0 movz x18, #18, #0         [        19]
[.....................fdn.pic......r.............................................]-(         120000) 0x004000fc.0 movz x19, #19, #0         [        20]</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain">24.22.4.8.5. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_gain</a></h6>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall</a> but now with an LDR stall: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/stall_gain.S">userland/arch/aarch64/freestanding/linux/stall_gain.S</a>.</p>
</div>
<div class="paragraph">
<p>So in this case we see that there were actual potential gains, since the <code>movz x11</code> started running immediately. We just stopped at <code>movz x20</code> because a new ifetch was needed.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x004000b0.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x004000b4.0 movz x1, #1, #0           [         2]
[.ic.r........................................................................fdn]-(          40000) 0x004000b8.0 movz x2, #4, #0           [         3]
[.ic.r........................................................................fdn]-(          40000) 0x004000bc.0 movz x3, #5, #0           [         4]
[....................fdn.ic.r....................................................]-(         120000) 0x004000c0.0 adr x4, #65772            [         5]
[....................fdn.pic.....................................................]-(         120000) 0x004000c4.0 ldr x5, [x4]              [         6]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000c8.0 movz x6, #6, #0           [         7]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000cc.0 movz x7, #7, #0           [         8]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000d0.0 movz x8, #8, #0           [         9]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000d4.0 movz x9, #9, #0           [        10]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000d8.0 movz x10, #10, #0         [        11]
[........................................................r.......................]-(         160000)     ...
[....................fdn.pic.....................................................]-(         120000) 0x004000dc.0 movz x11, #11, #0         [        12]
[........................................................r.......................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e0.0 movz x12, #12, #0         [        13]
[........................................................r.......................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e4.0 movz x13, #13, #0         [        14]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e8.0 movz x14, #14, #0         [        15]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000ec.0 movz x15, #15, #0         [        16]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000f0.0 movz x16, #16, #0         [        17]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.pic....................................................]-(         120000) 0x004000f4.0 movz x17, #17, #0         [        18]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.pic....................................................]-(         120000) 0x004000f8.0 movz x18, #18, #0         [        19]
[.........................................................r......................]-(         160000)     ...
[.....................fdn.pic....................................................]-(         120000) 0x004000fc.0 movz x19, #19, #0         [        20]
[.........................................................r......................]-(         160000)     ...
[............................................fdn.ic.......r......................]-(         160000) 0x00400100.0 movz x20, #20, #0         [        21]
[............................................fdn.ic........r.....................]-(         160000) 0x00400104.0 movz x21, #21, #0         [        22]
[............................................fdn.ic........r.....................]-(         160000) 0x00400108.0 movz x22, #22, #0         [        23]
[............................................fdn.ic........r.....................]-(         160000) 0x0040010c.0 movz x23, #23, #0         [        24]
[............................................fdn.ic........r.....................]-(         160000) 0x00400110.0 movz x24, #24, #0         [        25]
[............................................fdn.ic........r.....................]-(         160000) 0x00400114.0 movz x25, #25, #0         [        26]
[............................................fdn.pic.......r.....................]-(         160000) 0x00400118.0 movz x26, #26, #0         [        27]
[............................................fdn.pic.......r.....................]-(         160000) 0x0040011c.0 movz x27, #27, #0         [        28]
[.............................................fdn.ic.......r.....................]-(         160000) 0x00400120.0 movz x28, #28, #0         [        29]
[.............................................fdn.ic........r....................]-(         160000) 0x00400124.0 movz x29, #29, #0         [        30]
[.............................................fdn.ic........r....................]-(         160000) 0x00400128.0 movz x0, #0, #0           [        31]
[.............................................fdn.ic........r....................]-(         160000) 0x0040012c.0 movz x1, #1, #0           [        32]
[.............................................fdn.pic.......r....................]-(         160000) 0x00400130.0 movz x2, #2, #0           [        33]
[.............................................fdn.pic.......r....................]-(         160000) 0x00400134.0 movz x3, #3, #0           [        34]
[.............................................fdn.pic.......r....................]-(         160000) 0x00400138.0 movz x4, #4, #0           [        35]
[.............................................fdn.pic.......r....................]-(         160000) 0x0040013c.0 movz x5, #5, #0           [        36]</pre>
</div>
</div>
<div class="paragraph">
<p>We now also understand the graph better from lines such as this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[....................fdn.pic.....................................................]-(         120000) 0x004000c4.0 ldr x5, [x4]              [         6]
[........................................................r.......................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000c8.0 movz x6, #6, #0           [         7]
[........................................................r.......................]-(         160000)     ...</pre>
</div>
</div>
<div class="paragraph">
<p>We see that extra lines are drawn (the <code>160000 &#8230;&#8203; lines</code> here) whenever something stalls for a period longer than the width of the visualisation.</p>
</div>
<div class="paragraph">
<p>Things are still relatively readable because the wrapping aligns them with events that actually happened on that line directly e.g. <code>160000) 0x00400100.0 movz x20, #20, #0.</code>.</p>
</div>
<div class="paragraph">
<p>But from this we kind of see the need for: <a href="#gem5-konata-o3-pipeline-viewer">gem5 Konata O3 pipeline viewer</a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-hazard4"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-hazard4"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-hazard4">24.22.4.8.6. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_hazard4</a></h6>
<div class="paragraph">
<p>Like <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-stall-gain">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: stall_gain</a> but now with some dependencies after the LDR: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/stall_hazard4.S">userland/arch/aarch64/freestanding/linux/stall_hazard4.S</a>.</p>
</div>
<div class="paragraph">
<p>So in this case the <code>ic</code> of dependencies like <code>add x6, x5, #1</code> have to wait until the LDR is finished:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.ic.r........................................................................fdn]-(          40000) 0x004000b0.0 movz x0, #0, #0           [         1]
[.ic.r........................................................................fdn]-(          40000) 0x004000b4.0 movz x1, #1, #0           [         2]
[.ic.r........................................................................fdn]-(          40000) 0x004000b8.0 movz x2, #4, #0           [         3]
[.ic.r........................................................................fdn]-(          40000) 0x004000bc.0 movz x3, #5, #0           [         4]
[....................fdn.ic.r....................................................]-(         120000) 0x004000c0.0 adr x4, #65772            [         5]
[....................fdn.pic.....................................................]-(         120000) 0x004000c4.0 ldr x5, [x4]              [         6]
[........................................................r.......................]-(         160000)     ...
[....................fdn.p.......................................................]-(         120000) 0x004000c8.0 add x6, x5, #1            [         7]
[......................................................ic.r......................]-(         160000)     ...
[....................fdn.p.......................................................]-(         120000) 0x004000cc.0 add x7, x6, #1            [         8]
[.......................................................ic.r.....................]-(         160000)     ...
[....................fdn.p.......................................................]-(         120000) 0x004000d0.0 add x8, x7, #1            [         9]
[........................................................ic.r....................]-(         160000)     ...
[....................fdn.p.......................................................]-(         120000) 0x004000d4.0 add x9, x8, #1            [        10]
[.........................................................ic.r...................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000d8.0 movz x10, #10, #0         [        11]
[............................................................r...................]-(         160000)     ...
[....................fdn.ic......................................................]-(         120000) 0x004000dc.0 movz x11, #11, #0         [        12]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e0.0 movz x12, #12, #0         [        13]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e4.0 movz x13, #13, #0         [        14]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000e8.0 movz x14, #14, #0         [        15]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000ec.0 movz x15, #15, #0         [        16]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000f0.0 movz x16, #16, #0         [        17]
[............................................................r...................]-(         160000)     ...
[.....................fdn.ic.....................................................]-(         120000) 0x004000f4.0 movz x17, #17, #0         [        18]
[.............................................................r..................]-(         160000)     ...
[.....................fdn.pic....................................................]-(         120000) 0x004000f8.0 movz x18, #18, #0         [        19]
[.............................................................r..................]-(         160000)     ...
[.....................fdn.pic....................................................]-(         120000) 0x004000fc.0 movz x19, #19, #0         [        20]
[.............................................................r..................]-(         160000)     ...
[............................................fdn.ic...........r..................]-(         160000) 0x00400100.0 movz x20, #20, #0         [        21]
[............................................fdn.ic...........r..................]-(         160000) 0x00400104.0 movz x21, #21, #0         [        22]
[............................................fdn.ic...........r..................]-(         160000) 0x00400108.0 movz x22, #22, #0         [        23]
[............................................fdn.ic...........r..................]-(         160000) 0x0040010c.0 movz x23, #23, #0         [        24]
[............................................fdn.ic...........r..................]-(         160000) 0x00400110.0 movz x24, #24, #0         [        25]
[............................................fdn.ic............r.................]-(         160000) 0x00400114.0 movz x25, #25, #0         [        26]
[............................................fdn.pic...........r.................]-(         160000) 0x00400118.0 movz x26, #26, #0         [        27]
[............................................fdn.pic...........r.................]-(         160000) 0x0040011c.0 movz x27, #27, #0         [        28]
[.............................................fdn.ic...........r.................]-(         160000) 0x00400120.0 movz x28, #28, #0         [        29]
[.............................................fdn.ic...........r.................]-(         160000) 0x00400124.0 movz x29, #29, #0         [        30]
[.............................................fdn.ic...........r.................]-(         160000) 0x00400128.0 movz x0, #0, #0           [        31]
[.............................................fdn.ic...........r.................]-(         160000) 0x0040012c.0 movz x1, #1, #0           [        32]
[.............................................fdn.pic..........r.................]-(         160000) 0x00400130.0 movz x2, #2, #0           [        33]
[.............................................fdn.pic...........r................]-(         160000) 0x00400134.0 movz x3, #3, #0           [        34]
[.............................................fdn.pic...........r................]-(         160000) 0x00400138.0 movz x4, #4, #0           [        35]
[.............................................fdn.pic...........r................]-(         160000) 0x0040013c.0 movz x5, #5, #0           [        36]</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative"><a class="anchor" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative"></a><a class="link" href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative">24.22.4.8.7. gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative</a></h6>
<div class="paragraph">
<p>Now let&#8217;s try to see some <a href="#speculative-execution">Speculative execution</a> in action with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/speculative.S">userland/arch/aarch64/freestanding/linux/speculative.S</a>.</p>
</div>
<div class="paragraph">
<p>That program is setup such that the branch is not taken if an extra CLI argument is passed with <code>--cli-args</code>.</p>
</div>
<div class="paragraph">
<p>We purposefully set things up so that speculation will be running from the icache so we can see what is going on more clearly without ifetch stalls.</p>
</div>
<div class="paragraph">
<p>Without an extra CLI argument (the branch is taken):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.............................................................................fdn]-(          40000) 0x00400078.0 ldr x0, [sp]              [         1]
[.ic.............................................................................]-(          80000)     ...
[................................r...............................................]-(         120000)     ...
[.............................................................................fdn]-(          40000) 0x0040007c.0 movz x1, #1, #0           [         2]
[.ic.............................................................................]-(          80000)     ...
[................................r...............................................]-(         120000)     ...
[....................fdn.ic......r...............................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [         3]
[....................fdn.ic......r...............................................]-(         120000) 0x00400084.0 movz x3, #3, #0           [         4]
[....................fdn.ic......r...............................................]-(         120000) 0x00400088.0 movz x4, #4, #0           [         5]
[....................fdn.ic......r...............................................]-(         120000) 0x0040008c.0 movz x5, #5, #0           [         6]
[....................fdn.ic......r...............................................]-(         120000) 0x00400090.0 movz x6, #6, #0           [         7]
[....................fdn.p.....ic..r.............................................]-(         120000) 0x00400094.0 subs x0, #2               [         8]
[....................fdn.ic........r.............................................]-(         120000) 0x00400098.0 movz x0, #3, #0           [         9]
[....................fdn.p......ic.r.............................................]-(         120000) 0x0040009c.0 b.lt 0x400080             [        10]
[=====================fdn=ic=====================================================]-(         120000) 0x004000a0.0 -----movz x10, #10, #0    [        11]
[=====================fdn=ic=====================================================]-(         120000) 0x004000a4.0 -----movz x11, #11, #0    [        12]
[=====================fdn=ic=====================================================]-(         120000) 0x004000a8.0 -----movz x12, #12, #0    [        13]
[=====================fdn=ic=====================================================]-(         120000) 0x004000ac.0 -----movz x13, #13, #0    [        14]
[=====================fdn=ic=====================================================]-(         120000) 0x004000b0.0 -----movz x14, #14, #0    [        15]
[=====================fdn=ic=====================================================]-(         120000) 0x004000b4.0 -----movz x15, #15, #0    [        16]
[=====================fdn=pic====================================================]-(         120000) 0x004000b8.0 -----movz x16, #16, #0    [        17]
[=====================fdn=pic====================================================]-(         120000) 0x004000bc.0 -----movz x17, #17, #0    [        18]
[.....................................fdn.ic.r...................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [        19]
[.....................................fdn.ic.r...................................]-(         120000) 0x00400084.0 movz x3, #3, #0           [        20]
[.....................................fdn.ic.r...................................]-(         120000) 0x00400088.0 movz x4, #4, #0           [        21]
[.....................................fdn.ic.r...................................]-(         120000) 0x0040008c.0 movz x5, #5, #0           [        22]
[.....................................fdn.ic.r...................................]-(         120000) 0x00400090.0 movz x6, #6, #0           [        23]
[.....................................fdn.pic.r..................................]-(         120000) 0x00400098.0 movz x0, #3, #0           [        25]
[.....................................fdn.pic.r..................................]-(         120000) 0x0040009c.0 b.lt 0x400080             [        26]
[......................................fdn.ic.r..................................]-(         120000) 0x004000a0.0 movz x10, #10, #0         [        27]
[......................................fdn.ic.r..................................]-(         120000) 0x004000a4.0 movz x11, #11, #0         [        28]
[......................................fdn.ic.r..................................]-(         120000) 0x004000a8.0 movz x12, #12, #0         [        29]
[......................................fdn.ic.r..................................]-(         120000) 0x004000ac.0 movz x13, #13, #0         [        30]
[......................................fdn.pic.r.................................]-(         120000) 0x004000b0.0 movz x14, #14, #0         [        31]
[......................................fdn.pic.r.................................]-(         120000) 0x004000b4.0 movz x15, #15, #0         [        32]
[......................................fdn.pic.r.................................]-(         120000) 0x004000b8.0 movz x16, #16, #0         [        33]
[......................................fdn.pic.r.................................]-(         120000) 0x004000bc.0 movz x17, #17, #0         [        34]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000c0.0 movz x0, #0, #0           [        35]
[.............................................fdn.ic.r...........................]-(         160000) 0x004000c4.0 movz x8, #93, #0          [        36]</pre>
</div>
</div>
<div class="paragraph">
<p>So here we see that the CPU mispredicted! After the <a href="#arm-branch-instructions">BLT instruction</a>, the CPU continued to run <code>movz x10</code>, assuming that the branch would not be taken.</p>
</div>
<div class="paragraph">
<p>Then, at time 120000, the LDR data came back, after the wrong prediction had already been fully executed.</p>
</div>
<div class="paragraph">
<p>The CPU then noticed that it mispredicted, and so it started again from the correct branch target <code>movz x2</code>, and the instructions that were thrown away are marked as <code>=====</code> in the timeline.</p>
</div>
<div class="paragraph">
<p>We can also see some <a href="#branch-predictor">Branch predictor</a> log lines in the <code>O3CPUAll</code> log:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> 130000: Fetch: system.cpu.fetch: [tid:0] [sn:10] Branch at PC 0x40009c predicted to be not taken
 130000: Fetch: system.cpu.fetch: [tid:0] [sn:10] Branch at PC 0x40009c predicted to go to (0x4000a0=&gt;0x4000a4).(0=&gt;1)

 131500: Commit: system.cpu.commit: [tid:10] [sn:0] Inserting PC (0x40009c=&gt;0x4000a0).(0=&gt;1) into ROB.
 131500: ROB: system.cpu.rob: Adding inst PC (0x40009c=&gt;0x4000a0).(0=&gt;1) to the ROB.
 131500: ROB: system.cpu.rob: [tid:0] Now has 10 instructions.

 132000: IEW: system.cpu.iew: [tid:0] Issue: Adding PC (0x40009c=&gt;0x4000a0).(0=&gt;1) [sn:10] [tid:0] to IQ.
 132000: IQ: system.cpu.iq: Adding instruction [sn:10] PC (0x40009c=&gt;0x4000a0).(0=&gt;1) to the IQ.
 132000: IQ: system.cpu.iq: Instruction PC (0x40009c=&gt;0x4000a0).(0=&gt;1) has src reg 6 (CCRegClass) that is being added to the dependency chain.
 132000: IQ: system.cpu.iq: Instruction PC (0x40009c=&gt;0x4000a0).(0=&gt;1) has src reg 8 (CCRegClass) that is being added to the dependency chain.
 132000: IQ: system.cpu.iq: Instruction PC (0x40009c=&gt;0x4000a0).(0=&gt;1) has src reg 7 (CCRegClass) that is being added to the dependency chain.

 135500: IQ: system.cpu.iq: Waking up a dependent instruction, [sn:10] PC (0x40009c=&gt;0x4000a0).(0=&gt;1).
 135500: IQ: global: [sn:10] has 1 ready out of 3 sources. RTI 0)
 135500: IQ: system.cpu.iq: Waking any dependents on register 7 (CCRegClass).
 135500: IQ: system.cpu.iq: Waking up a dependent instruction, [sn:10] PC (0x40009c=&gt;0x4000a0).(0=&gt;1).
 135500: IQ: global: [sn:10] has 2 ready out of 3 sources. RTI 0)
 135500: IQ: system.cpu.iq: Waking any dependents on register 8 (CCRegClass).
 135500: IQ: system.cpu.iq: Waking up a dependent instruction, [sn:10] PC (0x40009c=&gt;0x4000a0).(0=&gt;1).
 135500: IQ: global: [sn:10] has 3 ready out of 3 sources. RTI 0)
 135500: IQ: system.cpu.iq: Instruction is ready to issue, putting it onto the ready list, PC (0x40009c=&gt;0x4000a0).(0=&gt;1) opclass:1 [sn:10].
 135500: IEW: system.cpu.iew: Setting Destination Register 6 (CCRegClass)
 135500: Scoreboard: system.cpu.scoreboard: Setting reg 6 (CCRegClass) as ready
 135500: IEW: system.cpu.iew: Setting Destination Register 7 (CCRegClass)
 135500: Scoreboard: system.cpu.scoreboard: Setting reg 7 (CCRegClass) as ready
 135500: IEW: system.cpu.iew: Setting Destination Register 8 (CCRegClass)
 135500: Scoreboard: system.cpu.scoreboard: Setting reg 8 (CCRegClass) as ready
 135500: IQ: system.cpu.iq: Attempting to schedule ready instructions from the IQ.
 135500: IQ: system.cpu.iq: Thread 0: Issuing instruction PC (0x40009c=&gt;0x4000a0).(0=&gt;1) [sn:10]

 136000: IEW: system.cpu.iew: Execute: Processing PC (0x40009c=&gt;0x4000a0).(0=&gt;1), [tid:0] [sn:10].
 136000: IEW: global: RegFile: Access to cc register 6, has data 0x2
 136000: IEW: global: RegFile: Access to cc register 8, has data 0
 136000: IEW: global: RegFile: Access to cc register 7, has data 0
 136000: IEW: system.cpu.iew: Current wb cycle: 0, width: 8, numInst: 0
wbActual:0
 136000: IEW: system.cpu.iew: [tid:0] [sn:10] Execute: Branch mispredict detected.
 136000: IEW: system.cpu.iew: [tid:0] [sn:10] Predicted target was PC: (0x4000a0=&gt;0x4000a4).(0=&gt;1)
 136000: IEW: system.cpu.iew: [tid:0] [sn:10] Execute: Redirecting fetch to PC: (0x40009c=&gt;0x400080).(0=&gt;1)
 136000: IEW: system.cpu.iew: [tid:0] [sn:10] Squashing from a specific instruction, PC: (0x40009c=&gt;0x400080).(0=&gt;1)

 136500: Commit: system.cpu.commit: [tid:0] Squashing due to branch mispred PC:0x40009c [sn:10]
 136500: Commit: system.cpu.commit: [tid:0] Redirecting to PC 0x400084
 136500: ROB: system.cpu.rob: Starting to squash within the ROB.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instructions until [sn:10].
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000bc=&gt;0x4000c0).(0=&gt;1), seq num 18.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000b8=&gt;0x4000bc).(0=&gt;1), seq num 17.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000b4=&gt;0x4000b8).(0=&gt;1), seq num 16.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000b0=&gt;0x4000b4).(0=&gt;1), seq num 15.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000ac=&gt;0x4000b0).(0=&gt;1), seq num 14.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000a8=&gt;0x4000ac).(0=&gt;1), seq num 13.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000a4=&gt;0x4000a8).(0=&gt;1), seq num 12.
 136500: ROB: system.cpu.rob: [tid:0] Squashing instruction PC (0x4000a0=&gt;0x4000a4).(0=&gt;1), seq num 11.
 136500: ROB: system.cpu.rob: [tid:0] Done squashing instructions.
 136500: Commit: system.cpu.commit: [tid:0] Marking PC (0x40009c=&gt;0x400080).(0=&gt;1), [sn:10] ready within ROB.

 137000: Commit: system.cpu.commit: [tid:0] [sn:10] Committing instruction with PC (0x40009c=&gt;0x400080).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+36    :   b.lt   0x400080          : IntAlu :   FetchSeq=10  CPSeq=10  flags=(IsControl|IsDirectControl|IsCondControl)
 137000: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x40009c=&gt;0x400080).(0=&gt;1), [sn:10]
 137000: O3CPU: system.cpu: Removing committed instruction [tid:0] PC (0x40009c=&gt;0x400080).(0=&gt;1) [sn:10]
 137000: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:11]
 137000: Commit: system.cpu.commit: Retiring squashed instruction from ROB.

 137000: Commit: system.cpu.commit: Trying to commit head instruction, [tid:0] [sn:10]
 137000: Commit: system.cpu.commit: [tid:0] [sn:10] Committing instruction with PC (0x40009c=&gt;0x400080).(0=&gt;1)
 130000: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+36    :   b.lt   0x400080          : IntAlu :   FetchSeq=10  CPSeq=10  flags=(IsControl|IsDirectControl|IsCondControl)

 138500: Fetch: system.cpu.fetch: [tid:0] [sn:26] Branch at PC 0x40009c predicted to be not taken
 138500: Fetch: system.cpu.fetch: [tid:0] [sn:26] Branch at PC 0x40009c predicted to go to (0x4000a0=&gt;0x4000a4).(0=&gt;1)

 142500: Commit: system.cpu.commit: [tid:0] [sn:26] Committing instruction with PC (0x40009c=&gt;0x4000a0).(0=&gt;1)
 138500: ExecEnable: system.cpu: A0 T0 : @asm_main_after_prologue+36    :   b.lt   0x400080          : IntAlu :   FetchSeq=26  CPSeq=18  flags=(IsControl|IsDirectControl|IsCondControl)
 142500: ROB: system.cpu.rob: [tid:0] Retiring head instruction, instruction PC (0x40009c=&gt;0x4000a0).(0=&gt;1), [sn:26]</pre>
</div>
</div>
<div class="paragraph">
<p>With an extra CLI (the branch is not taken):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// f = fetch, d = decode, n = rename, p = dispatch, i = issue, c = complete, r = retire

                                     timeline                                             tick          pc.upc     disasm                      seq_num
[.............................................................................fdn]-(          40000) 0x00400078.0 ldr x0, [sp]              [         1]
[.ic.............................................................................]-(          80000)     ...
[................................r...............................................]-(         120000)     ...
[.............................................................................fdn]-(          40000) 0x0040007c.0 movz x1, #1, #0           [         2]
[.ic.............................................................................]-(          80000)     ...
[................................r...............................................]-(         120000)     ...
[....................fdn.ic......r...............................................]-(         120000) 0x00400080.0 movz x2, #2, #0           [         3]
[....................fdn.ic......r...............................................]-(         120000) 0x00400084.0 movz x3, #3, #0           [         4]
[....................fdn.ic......r...............................................]-(         120000) 0x00400088.0 movz x4, #4, #0           [         5]
[....................fdn.ic......r...............................................]-(         120000) 0x0040008c.0 movz x5, #5, #0           [         6]
[....................fdn.ic......r...............................................]-(         120000) 0x00400090.0 movz x6, #6, #0           [         7]
[....................fdn.ic.......r..............................................]-(         120000) 0x00400098.0 movz x0, #3, #0           [         9]
[....................fdn.p......ic.r.............................................]-(         120000) 0x0040009c.0 b.lt 0x400080             [        10]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000a0.0 movz x10, #10, #0         [        11]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000a4.0 movz x11, #11, #0         [        12]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000a8.0 movz x12, #12, #0         [        13]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000ac.0 movz x13, #13, #0         [        14]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000b0.0 movz x14, #14, #0         [        15]
[.....................fdn.ic.......r.............................................]-(         120000) 0x004000b4.0 movz x15, #15, #0         [        16]
[.....................fdn.pic......r.............................................]-(         120000) 0x004000b8.0 movz x16, #16, #0         [        17]
[.....................fdn.pic.......r............................................]-(         120000) 0x004000bc.0 movz x17, #17, #0         [        18]
[............................................fdn.ic.r............................]-(         160000) 0x004000c0.0 movz x0, #0, #0           [        19]
[............................................fdn.ic.r............................]-(         160000) 0x004000c4.0 movz x8, #93, #0          [        20]</pre>
</div>
</div>
<div class="paragraph">
<p>So this time the prediction was correct. Retire is delayed until the memory comes back, but we otherwise just kept running forward until hitting the next ifetch cache line.</p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-instruction-definitions"><a class="anchor" href="#gem5-instruction-definitions"></a><a class="link" href="#gem5-instruction-definitions">24.22.5. gem5 instruction definitions</a></h4>
<div class="paragraph">
<p>This is one of the parts of gem5 that rely on semi-useless <a href="#gem5-code-generation">code generation</a> inside the <code>.isa</code> sublanguage.</p>
</div>
<div class="paragraph">
<p>Which is mostly Python, with some magic letters thrown in for good measure.</p>
</div>
<div class="paragraph">
<p>The class definitions get all dumped into one humongous C++ include file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/exec-ns.cc.inc</pre>
</div>
</div>
<div class="paragraph">
<p>That file defines the key methods of each instruction, e.g. the ARM immediate <a href="#userland-assembly">ADD instruction</a> has its <code>execute</code> method defined there:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Fault AddImm::execute(
        ExecContext *xc, Trace::InstRecord *traceData) const</pre>
</div>
</div>
<div class="paragraph">
<p>or for example the key methods of an <a href="#arm-str-instruction">ARM 64-bit (X) STR with an immediate offset</a> (<code>STR &lt;Wt&gt;, [&lt;Xn|SP&gt;], #&lt;simm&gt;</code>):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Fault STRX64_IMM::execute(ExecContext *xc,
                                  Trace::InstRecord *traceData) const

    Fault STRX64_IMM::initiateAcc(ExecContext *xc,
                                      Trace::InstRecord *traceData) const

    Fault STRX64_IMM::completeAcc(PacketPtr pkt, ExecContext *xc,
                                      Trace::InstRecord *traceData) const
    {
        return NoFault;
    }</pre>
</div>
</div>
<div class="paragraph">
<p>We also notice that the key argument passed to those instructions is of type <code>ExecContext</code>, which is discussed further at: <a href="#gem5-execcontext">Section 24.22.6.3, &#8220;gem5 <code>ExecContext</code>&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The file is an include so that compilation can be split up into chunks by the autogenerated includers</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/generic_cpu_1.cc
build/ARM/arch/arm/generated/generic_cpu_2.cc
...</pre>
</div>
</div>
<div class="paragraph">
<p>via the <code>__SPLIT</code> macro as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#include "exec-g.cc.inc"
#include "cpu/exec_context.hh"
#include "decoder.hh"
namespace ArmISAInst {
#define __SPLIT 1
#include "exec-ns.cc.inc"
}</pre>
</div>
</div>
<div class="paragraph">
<p>This is likely done to not overload the C++ compiler? But sure enough overloads IDEs and GDB which takes forever to load the source of any frames going through it.</p>
</div>
<div class="paragraph">
<p>We should split that file into one per class for the love of God.</p>
</div>
<div class="paragraph">
<p>The autogenerated instruction class declarations can be found at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/decoder-ns.hh.inc</pre>
</div>
</div>
<div class="paragraph">
<p>and the autogenerated bulk of the decoder:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/decoder-ns.cc.inc</pre>
</div>
</div>
<div class="paragraph">
<p>which also happens to contain the constructor definitions of the instruction classes, e.g. for the ADD immediate because why not:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    AddImm::AddImm(ExtMachInst machInst,
                                          IntRegIndex _dest,
                                          IntRegIndex _op1,
                                          uint32_t _imm,
                                          bool _rotC)</pre>
</div>
</div>
<div class="paragraph">
<p>The above files get tied in the autogenerated:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/decoder.hh</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#include "decoder-g.hh.inc"
namespace ArmISAInst {
#include "decoder-ns.hh.inc"
}</pre>
</div>
</div>
<div class="paragraph">
<p>Different instructions inherit form different classes, e.g. the ARM immediate ADD instruction is a <code>DataImmOp</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class AddImm : public DataImmOp
{
    public:
        // Constructor
        AddImm(ExtMachInst machInst, IntRegIndex _dest,
                IntRegIndex _op1, uint32_t _imm, bool _rotC=true);
        Fault execute(ExecContext *, Trace::InstRecord *) const override;
};</pre>
</div>
</div>
<div class="paragraph">
<p>and STRX64_IMM is an <code>ArmISA::MemoryImm64</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    class STRX64_IMM : public ArmISA::MemoryImm64
    {
      public:

        /// Constructor.
        STRX64_IMM(ExtMachInst machInst,
                IntRegIndex _dest, IntRegIndex _base, int64_t _imm);

        Fault execute(ExecContext *, Trace::InstRecord *) const override;
        Fault initiateAcc(ExecContext *, Trace::InstRecord *) const override;
        Fault completeAcc(PacketPtr, ExecContext *,
                          Trace::InstRecord *) const override;

        void
        annotateFault(ArmFault *fault) override
        {
                    fault-&gt;annotate(ArmFault::SAS, 3);
                    fault-&gt;annotate(ArmFault::SSE, false);
                    fault-&gt;annotate(ArmFault::SRT, dest);
                    fault-&gt;annotate(ArmFault::SF, true);
                    fault-&gt;annotate(ArmFault::AR, false);
        }
    };</pre>
</div>
</div>
<div class="paragraph">
<p>but different memory instructions can have different base classes too e.g. <a href="#arm-ldxr-and-stxr-instructions">STXR</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class STXRX64 : public ArmISA::MemoryEx64</pre>
</div>
</div>
<div class="paragraph">
<p>A summarized class hierarchy for the above is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>StaticInst</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::ArmStaticInst</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::PredOp</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::DataImmOp</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::AddImm</code></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>ArmISA::MightBeMicro64</code></p>
<div class="ulist">
<ul>
<li>
<p>ArmISA::Memory64</p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::MemoryImm64</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::MemoryEx64</code></p>
<div class="ulist">
<ul>
<li>
<p><code>ArmISA::STXRX64</code></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Tested in gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.</p>
</div>
<div class="sect4">
<h5 id="gem5-execute-vs-initiateacc-vs-completeacc"><a class="anchor" href="#gem5-execute-vs-initiateacc-vs-completeacc"></a><a class="link" href="#gem5-execute-vs-initiateacc-vs-completeacc">24.22.5.1. gem5 <code>execute</code> vs <code>initiateAcc</code> vs <code>completeAcc</code></a></h5>
<div class="paragraph">
<p>These are the key methods defined in instruction definitions, so lets see when each one gets called and what they do more or less.</p>
</div>
<div class="paragraph">
<p><code>execute</code> is the only one of the three that gets defined by "non-memory" instructions.</p>
</div>
<div class="paragraph">
<p>Memory instructions define all three.</p>
</div>
<div class="paragraph">
<p>The three methods are present in the base class <code>StaticInst</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    virtual Fault execute(ExecContext *xc,
                          Trace::InstRecord *traceData) const = 0;

    virtual Fault initiateAcc(ExecContext *xc,
                              Trace::InstRecord *traceData) const
    {
        panic("initiateAcc not defined!");
    }

    virtual Fault completeAcc(Packet *pkt, ExecContext *xc,
                              Trace::InstRecord *traceData) const
    {
        panic("completeAcc not defined!");
    }</pre>
</div>
</div>
<div class="paragraph">
<p>so we see that all instructions must implement <code>execute</code>, while overriding <code>initiateAcc</code> and <code>completeAcc</code> are optional and only done by classes for which those might get called: memory instructions.</p>
</div>
<div class="paragraph">
<p><code>execute</code> is what does the actual job for non-memory instructions (obviously, since it is the only one of the three methods that is defined as not <code>panic</code> for those).</p>
</div>
<div class="paragraph">
<p>Memory instructions however run either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>execute</code> in <code>AtomicSimpleCPU</code>: this does the entire memory access in one go</p>
</li>
<li>
<p><code>initiateAcc</code> + <code>completeAcc</code> in timing CPUs. <code>initiateAcc</code> is called when the instruction starts executing, and <code>completeAcc</code> is called when the memory fetch returns from the memory system.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This can be seen concretely in GDB from the analysis done at: <a href="#timingsimplecpu-analysis-ldr-stall">TimingSimpleCPU analysis: LDR stall</a> and for more memory details see <a href="#gem5-functional-vs-atomic-vs-timing-memory-requests">gem5 functional vs atomic vs timing memory requests</a>.</p>
</div>
<div class="sect5">
<h6 id="gem5-completeacc"><a class="anchor" href="#gem5-completeacc"></a><a class="link" href="#gem5-completeacc">24.22.5.1.1. gem5 <code>completeAcc</code></a></h6>
<div class="paragraph">
<p><code>completeAcc</code> is boring on most simple store memory instructions, e.g. a simple STR:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Fault STRX64_IMM::completeAcc(PacketPtr pkt, ExecContext *xc,
                                      Trace::InstRecord *traceData) const
    {
        return NoFault;
    }</pre>
</div>
</div>
<div class="paragraph">
<p>This is because the store does all of its job on <code>completeAcc</code> basically, creating the memory write request.</p>
</div>
<div class="paragraph">
<p>Loads however have non-trivial <code>completeAcc</code>, because now we have at the very least, to save the value read from memory into a CPU address.</p>
</div>
<div class="paragraph">
<p>Things are much more interesting however on more interesting loads, for example <a href="#arm-ldxr-and-stxr-instructions">STXR</a> (hand formatted here):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault STXRX64::completeAcc(PacketPtr pkt, ExecContext *xc,
                                    Trace::InstRecord *traceData) const {
    Fault fault = NoFault;
    uint64_t XResult = 0;
    uint32_t SevMailbox = 0;
    uint32_t LLSCLock = 0;
    uint64_t writeResult = pkt-&gt;req-&gt;getExtraData();
    XResult = !writeResult; SevMailbox = 1; LLSCLock = 0;
    if (fault == NoFault) {
        {
            uint64_t final_val = XResult;
            xc-&gt;setIntRegOperand(this, 0, (XResult) &amp; mask(aarch64 ? 64 : 32));
            if (traceData) { traceData-&gt;setData(final_val); }
        }
        xc-&gt;setMiscRegOperand(this, 1, SevMailbox);
        if (traceData) { traceData-&gt;setData(SevMailbox); }
        xc-&gt;setMiscRegOperand(this, 2, LLSCLock);
        if (traceData) { traceData-&gt;setData(LLSCLock); }
    }
    return fault;
}</pre>
</div>
</div>
<div class="paragraph">
<p>From GDB on <a href="#timingsimplecpu-analysis-ldr-stall">TimingSimpleCPU analysis: LDR stall</a> we see that <code>completeAcc</code> gets called from <code>TimingSimpleCPU::completeDataAccess</code>.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-microops"><a class="anchor" href="#gem5-microops"></a><a class="link" href="#gem5-microops">24.22.5.2. gem5 microops</a></h5>
<div class="paragraph">
<p>Some gem5 instructions break down into multiple microops.</p>
</div>
<div class="paragraph">
<p>Microops are very similar to regular instructions, and show on the <a href="#gem5-execall-trace-format">gem5 ExecAll trace format</a> since that flag implies <code>ExecMicro</code>.</p>
</div>
<div class="paragraph">
<p>On aarch64 for example, one of the simplest microoped instructions is <a href="#armv8-aarch64-ldp-and-stp-instructions">STP</a>, which does the relatively complex operation of storing two values to memory at once, and is therefore a good candidate for being broken down into microops. We can observe it when executing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch arch64 \
  --emulator gem5 \
  --trace-insts-stdout \
  --userland userland/arch/aarch64/freestanding/linux/disassembly_test.S \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which contains in gem5&#8217;s broken-ish disassembly that the input:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>stp x1, x2 [x0, 16]</pre>
</div>
</div>
<div class="paragraph">
<p>generated the output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  16500: system.cpu: A0 T0 : @_start+108    : stp
  16500: system.cpu: A0 T0 : @_start+108. 0 :   addxi_uop   ureg0, x0, #16 : IntAlu :  D=0x0000000000420010  flags=(IsInteger|IsMicroop|IsDelayedCommit|IsFirstMicroop)
  17000: system.cpu: A0 T0 : @_start+108. 1 :   strxi_uop   w1, [ureg0]  : MemWrite :  D=0x000000009abcdef0 A=0x420010  flags=(IsInteger|IsMemRef|IsStore|IsMicroop|IsDelayedCommit)
  17500: system.cpu: A0 T0 : @_start+108. 2 :   strxi_uop   w2, [ureg0, #8] : MemWrite :  D=0x0000000000000002 A=0x420018  flags=(IsInteger|IsMemRef|IsStore|IsMicroop|IsLastMicroop)</pre>
</div>
</div>
<div class="paragraph">
<p>Where <code>@_start+108. 0</code>, <code>@_start+108. 1</code> and <code>@_start+108. 2</code> all happen at the same PC, and are therefore microops of STP.</p>
</div>
<div class="paragraph">
<p>From their names, which are of course not specified in the <a href="#armarm8">ARMv8 architecture reference manual</a>, we guess that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>addxi_uop</code>: adds 16</p>
</li>
<li>
<p><code>strxi_uop</code>: stores one of the two members of the pair, like a regular STR</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>From the gem5 source code, we see that STP is a <code>class LdpStp : public PairMemOp</code>, and then the constructor of <code>PairMemOp</code> sets up the microops depending on the exact type of LDP/STP:</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-threadcontext-vs-threadstate-vs-execcontext-vs-process"><a class="anchor" href="#gem5-threadcontext-vs-threadstate-vs-execcontext-vs-process"></a><a class="link" href="#gem5-threadcontext-vs-threadstate-vs-execcontext-vs-process">24.22.6. gem5 <code>ThreadContext</code> vs <code>ThreadState</code> vs <code>ExecContext</code> vs <code>Process</code></a></h4>
<div class="paragraph">
<p>These classes get used everywhere, and they have a somewhat convoluted relation with one another, so let&#8217;s figure it out this mess.</p>
</div>
<div class="paragraph">
<p>None of those objects are <a href="#gem5-python-c-interaction">SimObjects</a>, so they must all belong to some higher SimObject.</p>
</div>
<div class="paragraph">
<p>This section and all children tested at gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.</p>
</div>
<div class="sect4">
<h5 id="gem5-threadcontext"><a class="anchor" href="#gem5-threadcontext"></a><a class="link" href="#gem5-threadcontext">24.22.6.1. gem5 <code>ThreadContext</code></a></h5>
<div class="paragraph">
<p>As we delve into more details below, we will reach the following conclusion: a <code>ThreadContext</code> represents on thread of a CPU with multiple <a href="#hardware-threads">Hardware threads</a>.</p>
</div>
<div class="paragraph">
<p>We therefore we can have multiple <code>ThreadContext</code> for each <a href="#gem5-cpu-types"><code>BaseCPU</code></a>.</p>
</div>
<div class="paragraph">
<p><code>ThreadContext</code> is what gets passed in syscalls, e.g.:</p>
</div>
<div class="paragraph">
<p>src/sim/syscall_emul.hh</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class OS&gt;
SyscallReturn
readFunc(SyscallDesc *desc, ThreadContext *tc,
        int tgt_fd, Addr buf_ptr, int nbytes)</pre>
</div>
</div>
<div class="paragraph">
<p>The class hierarchy for <code>ThreadContext</code> looks like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ThreadContext
  O3ThreadContext
  SimpleThread</pre>
</div>
</div>
<div class="paragraph">
<p>where the gem5 MinorCPU also uses <code>SimpleThread</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/** Minor will use the SimpleThread state for now */
typedef SimpleThread MinorThread;</pre>
</div>
</div>
<div class="paragraph">
<p>It is a bit confusing, things would be much clearer if <code>SimpleThread</code> was called instead <code>SimpleThreadContext</code>!</p>
</div>
<div class="paragraph">
<p><code>readIntReg</code> and other register access methods are some notable methods implemented in descendants, e.g. <a href="#gem5-simplethread"><code>SimpleThread::readIntReg</code></a>.</p>
</div>
<div class="paragraph">
<p>Essentially all methods of the base <code>ThreadContext</code> are pure virtual.</p>
</div>
<div class="sect5">
<h6 id="gem5-simplethread"><a class="anchor" href="#gem5-simplethread"></a><a class="link" href="#gem5-simplethread">24.22.6.1.1. gem5 <code>SimpleThread</code></a></h6>
<div class="paragraph">
<p><code>SimpleThread</code> storage defined on <a href="#gem5-basesimplecpu"><code>BaseSimpleCPU</code></a> for simple CPUs like <code>AtomicSimpleCPU</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    for (unsigned i = 0; i &lt; numThreads; i++) {
        if (FullSystem) {
            thread = new SimpleThread(this, i, p-&gt;system,
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        } else {
            thread = new SimpleThread(this, i, p-&gt;system, p-&gt;workload[i],
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        }
        threadInfo.push_back(new SimpleExecContext(this, thread));
        ThreadContext *tc = thread-&gt;getTC();
        threadContexts.push_back(tc);
    }</pre>
</div>
</div>
<div class="paragraph">
<p>and on <code>MinorCPU</code> for Minor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>MinorCPU::MinorCPU(MinorCPUParams *params) :
    BaseCPU(params),
    threadPolicy(params-&gt;threadPolicy)
{
    /* This is only written for one thread at the moment */
    Minor::MinorThread *thread;

    for (ThreadID i = 0; i &lt; numThreads; i++) {
        if (FullSystem) {
            thread = new Minor::MinorThread(this, i, params-&gt;system,
                    params-&gt;itb, params-&gt;dtb, params-&gt;isa[i]);
            thread-&gt;setStatus(ThreadContext::Halted);
        } else {
            thread = new Minor::MinorThread(this, i, params-&gt;system,
                    params-&gt;workload[i], params-&gt;itb, params-&gt;dtb,
                    params-&gt;isa[i]);
        }

        threads.push_back(thread);
        ThreadContext *tc = thread-&gt;getTC();
        threadContexts.push_back(tc);
    }</pre>
</div>
</div>
<div class="paragraph">
<p>Those are used from <a href="#gem5-execcontext">gem5 <code>ExecContext</code></a>.</p>
</div>
<div class="paragraph">
<p>From this we see that one CPU can have multiple threads, and that this is controlled from the Python:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BaseCPU::BaseCPU(Params *p, bool is_checker)
    : numThreads(p-&gt;numThreads)</pre>
</div>
</div>
<div class="paragraph">
<p>and since <code>SimpleThread</code> contains its registers, this must represent <a href="#hardware-threads">Hardware threads</a>.</p>
</div>
<div class="paragraph">
<p>If we analyse <code>SimpleThread::readIntReg</code>, we see that the actual register data is contained inside <code>ThreadContext</code> descendants, e.g. in <code>SimpleThread</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    RegVal
    readIntReg(RegIndex reg_idx) const override
    {
        int flatIndex = isa-&gt;flattenIntIndex(reg_idx);
        assert(flatIndex &lt; TheISA::NumIntRegs);
        uint64_t regVal(readIntRegFlat(flatIndex));
        DPRINTF(IntRegs, "Reading int reg %d (%d) as %#x.\n",
                reg_idx, flatIndex, regVal);
        return regVal;
    }

    RegVal readIntRegFlat(RegIndex idx) const override { return intRegs[idx]; }
    void
    setIntRegFlat(RegIndex idx, RegVal val) override
    {
        intRegs[idx] = val;
    }

    std::array&lt;RegVal, TheISA::NumIntRegs&gt; intRegs;</pre>
</div>
</div>
<div class="paragraph">
<p>Another notable type of method contained in <code>Thread</code> context are methods that forward to <a href="#gem5-threadstate">gem5 <code>ThreadState</code></a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-o3threadcontext"><a class="anchor" href="#gem5-o3threadcontext"></a><a class="link" href="#gem5-o3threadcontext">24.22.6.1.2. gem5 <code>O3ThreadContext</code></a></h6>
<div class="paragraph">
<p>Instantiation happens in the <code>FullO3CPU</code> constructor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>FullO3CPU&lt;Impl&gt;::FullO3CPU(DerivO3CPUParams *params)

    for (ThreadID tid = 0; tid &lt; this-&gt;numThreads; ++tid) {
        if (FullSystem) {
            // SMT is not supported in FS mode yet.
            assert(this-&gt;numThreads == 1);
            this-&gt;thread[tid] = new Thread(this, 0, NULL);

        // Setup the TC that will serve as the interface to the threads/CPU.
        O3ThreadContext&lt;Impl&gt; *o3_tc = new O3ThreadContext&lt;Impl&gt;;</pre>
</div>
</div>
<div class="paragraph">
<p>and the SimObject <code>DerivO3CPU</code> is just a <code>FullO3CPU</code> instantiation:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class DerivO3CPU : public FullO3CPU&lt;O3CPUImpl&gt;</pre>
</div>
</div>
<div class="paragraph">
<p><code>O3ThreadContext</code> is a template class:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class Impl&gt;
class O3ThreadContext : public ThreadContext</pre>
</div>
</div>
<div class="paragraph">
<p>The only <code>Impl</code> used appears to be <code>O3CPUImpl</code>? This is explicitly instantiated in the source:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template class O3ThreadContext&lt;O3CPUImpl&gt;;</pre>
</div>
</div>
<div class="paragraph">
<p>see also: <a href="https://stackoverflow.com/questions/64420547/in-gem5-how-do-i-know-the-specific-location-of-the-class/64423633#64423633" class="bare">https://stackoverflow.com/questions/64420547/in-gem5-how-do-i-know-the-specific-location-of-the-class/64423633#64423633</a></p>
</div>
<div class="paragraph">
<p>Unlike in <code>SimpleThread</code> however, <code>O3ThreadContext</code> does not contain the register data itself, e.g. <code>O3ThreadContext::readIntRegFlat</code> instead forwards to <code>cpu</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class Impl&gt;
RegVal
O3ThreadContext&lt;Impl&gt;::readIntRegFlat(RegIndex reg_idx) const
{
    return cpu-&gt;readArchIntReg(reg_idx, thread-&gt;threadId());
}</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    typedef typename Impl::O3CPU O3CPU;

   /** Pointer to the CPU. */
    O3CPU *cpu;</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>struct O3CPUImpl
{
    /** The O3CPU type to be used. */
    typedef FullO3CPU&lt;O3CPUImpl&gt; O3CPU;</pre>
</div>
</div>
<div class="paragraph">
<p>and at long last <code>FullO3CPU</code> contains the register values:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class Impl&gt;
RegVal
FullO3CPU&lt;Impl&gt;::readArchIntReg(int reg_idx, ThreadID tid)
{
    intRegfileReads++;
    PhysRegIdPtr phys_reg = commitRenameMap[tid].lookup(
            RegId(IntRegClass, reg_idx));

    return regFile.readIntReg(phys_reg);
}</pre>
</div>
</div>
<div class="paragraph">
<p>So we guess that this difference from <code>SimpleThread</code> is due to register renaming of the out of order implementation.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-threadstate"><a class="anchor" href="#gem5-threadstate"></a><a class="link" href="#gem5-threadstate">24.22.6.2. gem5 <code>ThreadState</code></a></h5>
<div class="paragraph">
<p>Owned one per <code>ThreadContext</code>.</p>
</div>
<div class="paragraph">
<p>Many <code>ThreadContext</code> methods simply forward to <code>ThreadState</code> implementations.</p>
</div>
<div class="paragraph">
<p><a href="#gem5-simplethread"><code>SimpleThread</code></a> inherits from <code>ThreadState</code>, and forwards to it on several methods e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    int cpuId() const override { return ThreadState::cpuId(); }
    uint32_t socketId() const override { return ThreadState::socketId(); }
    int threadId() const override { return ThreadState::threadId(); }
    void setThreadId(int id) override { ThreadState::setThreadId(id); }
    ContextID contextId() const override { return ThreadState::contextId(); }
    void setContextId(ContextID id) override { ThreadState::setContextId(id); }</pre>
</div>
</div>
<div class="paragraph">
<p><code>O3ThreadContext</code> on the other hand contains an <code>O3ThreadState</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class Impl&gt;
struct O3ThreadState : public ThreadState</pre>
</div>
</div>
<div class="paragraph">
<p>at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class Impl&gt;
class O3ThreadContext : public ThreadContext
{
    O3ThreadState&lt;Impl&gt; *thread

    ContextID contextId() const override { return thread-&gt;contextId(); }

    void setContextId(ContextID id) override { thread-&gt;setContextId(id); }</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-execcontext"><a class="anchor" href="#gem5-execcontext"></a><a class="link" href="#gem5-execcontext">24.22.6.3. gem5 <code>ExecContext</code></a></h5>
<div class="paragraph">
<p><code>ExecContext</code> gets used in <a href="#gem5-instruction-definitions">gem5 instruction definitions</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/arm/generated/exec-ns.cc.inc</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Fault Mul::execute(
        ExecContext *xc, Trace::InstRecord *traceData) const</pre>
</div>
</div>
<div class="paragraph">
<p>It contains methods to allow interacting with CPU state from inside instruction execution, notably reading and writing from/to registers.</p>
</div>
<div class="paragraph">
<p>For example, the ARM <code>mul</code> instruction uses <code>ExecContext</code> to read the input operands, multiply them, and write to the output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    Fault Mul::execute(
        ExecContext *xc, Trace::InstRecord *traceData) const
    {
        Fault fault = NoFault;
        uint64_t resTemp = 0;
        resTemp = resTemp;
        uint32_t OptCondCodesNZ = 0;
        uint32_t OptCondCodesC = 0;
        uint32_t OptCondCodesV = 0;
        uint32_t Reg0 = 0;
        uint32_t Reg1 = 0;
        uint32_t Reg2 = 0;

        OptCondCodesNZ = xc-&gt;readCCRegOperand(this, 0);
        OptCondCodesC = xc-&gt;readCCRegOperand(this, 1);
        OptCondCodesV = xc-&gt;readCCRegOperand(this, 2);
        Reg1 =
            ((reg1 == PCReg) ? readPC(xc) : xc-&gt;readIntRegOperand(this, 3));
        Reg2 =
            ((reg2 == PCReg) ? readPC(xc) : xc-&gt;readIntRegOperand(this, 4));

        if (testPredicate(OptCondCodesNZ, OptCondCodesC, OptCondCodesV, condCode)/*auto*/)
        {
            Reg0 = resTemp = Reg1 * Reg2;;
            if (fault == NoFault) {
                {
                    uint32_t final_val = Reg0;
                    ((reg0 == PCReg) ? setNextPC(xc, Reg0) : xc-&gt;setIntRegOperand(this, 0, Reg0));
                    if (traceData) { traceData-&gt;setData(final_val); }
                };
            }
        } else {
            xc-&gt;setPredicate(false);
        }

        return fault;
    }</pre>
</div>
</div>
<div class="paragraph">
<p><code>ExecContext</code> is however basically just a wrapper that forwards to other classes that actually contain the data in a microarchitectural neutral manner. For example, in <code>SimpleExecContext</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    /** Reads an integer register. */
    RegVal
    readIntRegOperand(const StaticInst *si, int idx) override
    {
        numIntRegReads++;
        const RegId&amp; reg = si-&gt;srcRegIdx(idx);
        assert(reg.isIntReg());
        return thread-&gt;readIntReg(reg.index());
    }</pre>
</div>
</div>
<div class="paragraph">
<p>So we see that this just does some register position bookkeeping needed for instruction execution, but the actual data comes from <a href="#gem5-simplethread"><code>SimpleThread::readIntReg</code></a>, which is a specialization of <a href="#gem5-threadcontext">gem5 <code>ThreadContext</code></a>.</p>
</div>
<div class="paragraph">
<p><code>ExecContext</code> is a fully virtual class. The hierarchy is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>ExecContext</code></p>
<div class="ulist">
<ul>
<li>
<p><code>SimpleExecContext</code></p>
</li>
<li>
<p><code>Minor::MinorExecContext</code></p>
</li>
<li>
<p><code>BaseDynInst</code></p>
<div class="ulist">
<ul>
<li>
<p><code>BaseO3DynInst</code></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>If we follow <code>SimpleExecContext</code> creation for example, we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class BaseSimpleCPU : public BaseCPU
{
    std::vector&lt;SimpleExecContext*&gt; threadInfo;</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BaseSimpleCPU::BaseSimpleCPU(BaseSimpleCPUParams *p)
    : BaseCPU(p),
      curThread(0),
      branchPred(p-&gt;branchPred),
      traceData(NULL),
      inst(),
      _status(Idle)
{
    SimpleThread *thread;

    for (unsigned i = 0; i &lt; numThreads; i++) {
        if (FullSystem) {
            thread = new SimpleThread(this, i, p-&gt;system,
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        } else {
            thread = new SimpleThread(this, i, p-&gt;system, p-&gt;workload[i],
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        }
        threadInfo.push_back(new SimpleExecContext(this, thread));
        ThreadContext *tc = thread-&gt;getTC();
        threadContexts.push_back(tc);
    }</pre>
</div>
</div>
<div class="paragraph">
<p>therefore there is one <code>ExecContext</code> for each <code>ThreadContext</code>, and each <code>ExecContext</code> knows about its own <code>ThreadContext</code>.</p>
</div>
<div class="paragraph">
<p>This makes sense, since each <code>ThreadContext</code> represents one CPU register set, and therefore needs a separate <code>ExecContext</code> which allows instruction implementations to access those registers.</p>
</div>
<div class="sect5">
<h6 id="gem5-execcontext-readintregoperand-register-resolution"><a class="anchor" href="#gem5-execcontext-readintregoperand-register-resolution"></a><a class="link" href="#gem5-execcontext-readintregoperand-register-resolution">24.22.6.3.1. gem5 <code>ExecContext::readIntRegOperand</code> register resolution</a></h6>
<div class="paragraph">
<p>Let&#8217;s have a look at how <code>ExecContext::readIntRegOperand</code> actually matches registers to decoded registers IDs, since it is not obvious.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s study a simple aarch64 register register addition:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>add x0, x1, x2</pre>
</div>
</div>
<div class="paragraph">
<p>which corresponds to the <code>AddXSReg</code> instruction (formatted and simplified):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault AddXSReg::execute(ExecContext *xc, Trace::InstRecord *traceData) const {
    uint64_t Op264 = 0;
    uint64_t Dest64 = 0;
    uint64_t Op164 = 0;
    Op264 = ((xc-&gt;readIntRegOperand(this, 0)) &amp; mask(intWidth));
    Op164 = ((xc-&gt;readIntRegOperand(this, 1)) &amp; mask(intWidth));
    uint64_t secOp = shiftReg64(Op264, shiftAmt, shiftType, intWidth);
    Dest64 = Op164 + secOp;
    uint64_t final_val = Dest64;
    xc-&gt;setIntRegOperand(this, 0, (Dest64) &amp; mask(intWidth));
    if (traceData) { traceData-&gt;setData(final_val); }
    return NoFault;
}</pre>
</div>
</div>
<div class="paragraph">
<p>So what are those magic <code>0</code> and <code>1</code> constants on <code>xc&#8594;readIntRegOperand(this, 0)</code> and <code>xc&#8594;readIntRegOperand(this, 1)</code>?</p>
</div>
<div class="paragraph">
<p>First, we guess that they  must be related to the reading of <code>x1</code> and <code>x2</code>, which are the inputs of the addition.</p>
</div>
<div class="paragraph">
<p>Next, we also guess that the <code>0</code> read must correspond to <code>x2</code>, since it later gets potentially shifted as mentioned at <a href="#arm-shift-suffixes">Section 30.4.4.1, &#8220;ARM shift suffixes&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s also have a look at the decoder code that builds the instruction instance in <code>build/ARM/arch/arm/generated/decoder-ns.cc.inc</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ArmShiftType type =
    (ArmShiftType)(uint8_t)bits(machInst, 23, 22);
if (type == ROR)
    return new Unknown64(machInst);
uint8_t imm6 = bits(machInst, 15, 10);
if (!bits(machInst, 31) &amp;&amp; bits(imm6, 5))
    return new Unknown64(machInst);
IntRegIndex rd = (IntRegIndex)(uint8_t)bits(machInst, 4, 0);
IntRegIndex rdzr = makeZero(rd);
IntRegIndex rn = (IntRegIndex)(uint8_t)bits(machInst, 9, 5);
IntRegIndex rm = (IntRegIndex)(uint8_t)bits(machInst, 20, 16);

return new AddXSReg(machInst, rdzr, rn, rm, imm6, type);</pre>
</div>
</div>
<div class="paragraph">
<p>and the ARM instruction pseudocode from the <a href="#armarm8">ARMv8 architecture reference manual</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ADD &lt;Xd&gt;, &lt;Xn&gt;, &lt;Xm&gt;{, &lt;shift&gt; #&lt;amount&gt;}</pre>
</div>
</div>
<div class="paragraph">
<p>and the constructor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>AddXSReg::AddXSReg(ExtMachInst machInst,
    IntRegIndex _dest,
    IntRegIndex _op1,
    IntRegIndex _op2,
    int32_t _shiftAmt,
    ArmShiftType _shiftType
) : DataXSRegOp("add", machInst, IntAluOp,
                _dest, _op1, _op2, _shiftAmt, _shiftType) {
    _numSrcRegs = 0;
    _numDestRegs = 0;
    _numFPDestRegs = 0;
    _numVecDestRegs = 0;
    _numVecElemDestRegs = 0;
    _numVecPredDestRegs = 0;
    _numIntDestRegs = 0;
    _numCCDestRegs = 0;
    _srcRegIdx[_numSrcRegs++] = RegId(IntRegClass, op2);
    _destRegIdx[_numDestRegs++] = RegId(IntRegClass, dest);
    _numIntDestRegs++;
    _srcRegIdx[_numSrcRegs++] = RegId(IntRegClass, op1);
    flags[IsInteger] = true;;
}</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>RegId</code> is just a container class, and so the lines that we care about for now are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>_srcRegIdx[_numSrcRegs++] = RegId(IntRegClass, op2);
_srcRegIdx[_numSrcRegs++] = RegId(IntRegClass, op1);</pre>
</div>
</div>
<div class="paragraph">
<p>which matches the guess we made earlier: <code>op2</code> is <code>0</code> and <code>op1</code> is <code>1</code> (<code>op1</code> and <code>op2</code> are the same as <code>_op1</code> and <code>_op2</code> which are set in the base constructor <code>DataXSRegOp</code>).</p>
</div>
<div class="paragraph">
<p>We also note that the register decodings (which the ARM spec says are <code>1</code> for <code>x1</code> and <code>2</code> for <code>x2</code>) are actually passed as enum <code>IntRegIndex</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    IntRegIndex _op1,
    IntRegIndex _op2,</pre>
</div>
</div>
<div class="paragraph">
<p>which are defined at <code>src/arch/arm/interegs.hh</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>enum IntRegIndex
{
    /* All the unique register indices. */
    INTREG_R0,
    INTREG_R1,
    INTREG_R2,</pre>
</div>
</div>
<div class="paragraph">
<p>Then <code>SimpleExecContext::readIntRegOperand</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    /** Reads an integer register. */
    RegVal
    readIntRegOperand(const StaticInst *si, int idx) override
    {
        numIntRegReads++;
        const RegId&amp; reg = si-&gt;srcRegIdx(idx);
        assert(reg.isIntReg());
        return thread-&gt;readIntReg(reg.index());
    }</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>const RegId&amp; srcRegIdx(int i)  const { return _srcRegIdx[i]; }</pre>
</div>
</div>
<div class="paragraph">
<p>which is what is populated in the constructor.</p>
</div>
<div class="paragraph">
<p>Then, <code>RegIndex::index() { return regIdx; }</code> just returns the decoded register bytes, and now <code>SimpleThread::readIntReg</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>RegVal readIntReg(RegIndex reg_idx) const override {
    int flatIndex = isa-&gt;flattenIntIndex(reg_idx);
    return readIntRegFlat(flatIndex);
}</pre>
</div>
</div>
<div class="paragraph">
<p><code>readIntRegFlag</code> is what finally reads from the int register array:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>RegVal SimpleThreadContext::readIntRegFlat(RegIndex idx) const override { return intRegs[idx]; }

std::array&lt;RegVal, TheISA::NumIntRegs&gt; SimpleThreadContext::intRegs;</pre>
</div>
</div>
<div class="paragraph">
<p>and then there is the flattening magic at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>int
flattenIntIndex(int reg) const
{
    assert(reg &gt;= 0);
    if (reg &lt; NUM_ARCH_INTREGS) {
        return intRegMap[reg];
    } else if (reg &lt; NUM_INTREGS) {
        return reg;
    } else if (reg == INTREG_SPX) {
        CPSR cpsr = miscRegs[MISCREG_CPSR];
        ExceptionLevel el = opModeToEL(
            (OperatingMode) (uint8_t) cpsr.mode);
        if (!cpsr.sp &amp;&amp; el != EL0)
            return INTREG_SP0;
        switch (el) {
            case EL3:
            return INTREG_SP3;
            case EL2:
            return INTREG_SP2;
            case EL1:
            return INTREG_SP1;
            case EL0:
            return INTREG_SP0;
            default:
            panic("Invalid exception level");
            return 0;  // Never happens.
        }
    } else {
        return flattenIntRegModeIndex(reg);
    }
}</pre>
</div>
</div>
<div class="paragraph">
<p>Then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    NUM_ARCH_INTREGS = 32,</pre>
</div>
</div>
<div class="paragraph">
<p>so we undertand that this covers x0 to x31. <code>NUM_INTREGS</code> is also 32, so I&#8217;m a bit confused, that case is never reached.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    INTREG_SPX = NUM_INTREGS,</pre>
</div>
</div>
<div class="paragraph">
<p>SP is 32, but it is a bit more magic, since in ARM there is one SP per <a href="#arm-exception-levels">exception level</a> as mentioned at <a href="#arm-sp0-vs-spx">ARM SP0 vs SPx</a>.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   INTREG_SPX = NUM_INTREGS</pre>
</div>
</div>
<div class="paragraph">
<p>We can also have a quick look at the <code>AddXImm</code> instruction which corresponds to a simple addition of an immediate as shown in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/add.S">userland/arch/aarch64/add.S</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>add x0, x1, 2</pre>
</div>
</div>
<div class="paragraph">
<p>Its <a href="#gem5-execute-vs-initiateacc-vs-completeacc"><code>execute</code> method</a> contains in <code>build/ARM/arch/arm/generated/exec-ns.cc.inc</code> (hand formatted and slightly simplified):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault AddXImm::execute(ExecContext *xc, Trace::InstRecord *traceData) const {
    uint64_t Dest64 = 0;
    uint64_t Op164 = 0;
    Op164 = ((xc-&gt;readIntRegOperand(this, 0)) &amp; mask(intWidth));
    Dest64 = Op164 + imm;
    uint64_t final_val = Dest64;
    xc-&gt;setIntRegOperand(this, 0, (Dest64) &amp; mask(intWidth));
    if (traceData) { traceData-&gt;setData(final_val); }
    return NoFault;
}</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>imm</code> is set directly on the constructor.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-process"><a class="anchor" href="#gem5-process"></a><a class="link" href="#gem5-process">24.22.6.4. gem5 <code>Process</code></a></h5>
<div class="paragraph">
<p>The <code>Process</code> class is used only for <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a>, and it represents a process like a Linux userland process, in addition to any further gem5 specific data needed to represent the process.</p>
</div>
<div class="paragraph">
<p>The first thing most syscall implementations do is to actually pull <code>Process</code> out of <a href="#gem5-threadcontext">gem5 <code>ThreadContext</code></a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>template &lt;class OS&gt;
SyscallReturn
readFunc(SyscallDesc *desc, ThreadContext *tc,
        int tgt_fd, Addr buf_ptr, int nbytes)
{
    auto p = tc-&gt;getProcessPtr();</pre>
</div>
</div>
<div class="paragraph">
<p>For example, we can readily see from its interface that it contains several accessors for common process fields:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    inline uint64_t uid() { return _uid; }
    inline uint64_t euid() { return _euid; }
    inline uint64_t gid() { return _gid; }
    inline uint64_t egid() { return _egid; }</pre>
</div>
</div>
<div class="paragraph">
<p><code>Process</code> is a <a href="#gem5-python-c-interaction"><code>SimObject</code></a>, and therefore produced directly in e.g. se.py.</p>
</div>
<div class="paragraph">
<p>se.py produces one process <a href="#gem5-syscall-emulation-multiple-executables">per-executable given</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    workloads = options.cmd.split(';')
    idx = 0
    for wrkld in workloads:
        process = Process(pid = 100 + idx)</pre>
</div>
</div>
<div class="paragraph">
<p>and those are placed in the <code>workload</code> property:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>for i in range(np):
    if options.smt:
        system.cpu[i].workload = multiprocesses
    elif len(multiprocesses) == 1:
        system.cpu[i].workload = multiprocesses[0]
    else:
        system.cpu[i].workload = multiprocesses[i]</pre>
</div>
</div>
<div class="paragraph">
<p>and finally each thread of a CPU gets assigned to a different such workload:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BaseSimpleCPU::BaseSimpleCPU(BaseSimpleCPUParams *p)
    : BaseCPU(p),
      curThread(0),
      branchPred(p-&gt;branchPred),
      traceData(NULL),
      inst(),
      _status(Idle)
{
    SimpleThread *thread;

    for (unsigned i = 0; i &lt; numThreads; i++) {
        if (FullSystem) {
            thread = new SimpleThread(this, i, p-&gt;system,
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        } else {
            thread = new SimpleThread(this, i, p-&gt;system, p-&gt;workload[i],
                                      p-&gt;itb, p-&gt;dtb, p-&gt;isa[i]);
        }
        threadInfo.push_back(new SimpleExecContext(this, thread));
        ThreadContext *tc = thread-&gt;getTC();
        threadContexts.push_back(tc);
    }</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-functional-units"><a class="anchor" href="#gem5-functional-units"></a><a class="link" href="#gem5-functional-units">24.22.7. gem5 functional units</a></h4>
<div class="paragraph">
<p>Each instruction is marked with a class, and each class can execute in a given <a href="#execution-unit">functional unit</a>.</p>
</div>
<div class="sect4">
<h5 id="gem5-minorcpu-default-functional-units"><a class="anchor" href="#gem5-minorcpu-default-functional-units"></a><a class="link" href="#gem5-minorcpu-default-functional-units">24.22.7.1. gem5 <code>MinorCPU</code> default functional units</a></h5>
<div class="paragraph">
<p>Which units are available is visible for example on the <a href="#gem5-config-ini">gem5 config.ini</a> of a <a href="#gem5-minorcpu">gem5 MinorCPU</a> run. Functional units are not present in simple CPUs like <a href="#gem5-timingsimplecpu">gem5 <code>TimingSimpleCPU</code></a>.</p>
</div>
<div class="paragraph">
<p>For example, on gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1, the <code>config.ini</code> of a minor run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run   \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace-insts-stdout \
  -- \
  --cpu-type MinorCPU \
  --caches</pre>
</div>
</div>
<div class="paragraph">
<p>contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu]
type=MinorCPU
children=branchPred dcache dtb executeFuncUnits icache interrupts isa itb power_state tracer workload
executeInputWidth=2
executeIssueLimit=2</pre>
</div>
</div>
<div class="paragraph">
<p>Here also note the <code>executeInputWidth=2</code> and <code>executeIssueLimit=2</code> suggesting that this is a <a href="#superscalar-processor">dual issue superscalar processor</a>.</p>
</div>
<div class="paragraph">
<p>The <code>system.cpu</code> points to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.executeFuncUnits]
type=MinorFUPool
children=funcUnits0 funcUnits1 funcUnits2 funcUnits3 funcUnits4 funcUnits5 funcUnits6 funcUnits7</pre>
</div>
</div>
<div class="paragraph">
<p>and the two first units are in full:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.executeFuncUnits.funcUnits0]
type=MinorFU
children=opClasses timings
opClasses=system.cpu.executeFuncUnits.funcUnits0.opClasses
opLat=3

[system.cpu.executeFuncUnits.funcUnits0.opClasses]
type=MinorOpClassSet
children=opClasses

[system.cpu.executeFuncUnits.funcUnits0.opClasses.opClasses]
type=MinorOpClass
opClass=IntAlu</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.executeFuncUnits.funcUnits1]
type=MinorFU
children=opClasses timings
opLat=3

[system.cpu.executeFuncUnits.funcUnits1.opClasses]
type=MinorOpClassSet
children=opClasses
opClasses=system.cpu.executeFuncUnits.funcUnits1.opClasses.opClasses

[system.cpu.executeFuncUnits.funcUnits1.opClasses.opClasses]
type=MinorOpClass
opClass=IntAlu</pre>
</div>
</div>
<div class="paragraph">
<p>So we understand that both:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the first and second functional units are <code>IntAlu</code>, so doing integer arithmetic operations</p>
</li>
<li>
<p>both have a latency of 3</p>
</li>
<li>
<p>each functional unit can have a set of <code>opClass</code> with more than one type. Those first two units just happen to have a single type.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The full list is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>0, 1: <code>IntAlu</code>, <code>opLat=3</code></p>
</li>
<li>
<p>2: <code>IntMult</code>, <code>opLat=3</code></p>
</li>
<li>
<p>3: <code>IntDiv</code>, <code>opLat=9</code>. So we see that a more complex operation such as division has higher latency.</p>
</li>
<li>
<p>4: <code>FloatAdd</code>, <code>FloatCmp</code>, and a gazillion other floating point related things. <code>opLat=6</code>.</p>
</li>
<li>
<p>5: <code>SimdPredAlu</code>: TODO SVE-related? <code>opLat=3</code></p>
</li>
<li>
<p>6: <code>MemRead</code>, <code>MemWrite</code>, <code>FloatMemRead</code>, <code>FloatMemWrite</code>. <code>opLat=1</code></p>
</li>
<li>
<p>7: <code>IprAccess</code> (TODO), <code>InstPrefetch</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These are of course all specified in <a href="#gem5-python-c-interaction">from the Python</a> at <code>src/cpu/minor/MinorCPU.py</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>class MinorDefaultFUPool(MinorFUPool):
    funcUnits = [MinorDefaultIntFU(), MinorDefaultIntFU(),
        MinorDefaultIntMulFU(), MinorDefaultIntDivFU(),
        MinorDefaultFloatSimdFU(), MinorDefaultPredFU(),
        MinorDefaultMemFU(), MinorDefaultMiscFU()]</pre>
</div>
</div>
<div class="paragraph">
<p>We then expect that each instruction has a certain <code>opClass</code> that determines on which unit it can run.</p>
</div>
<div class="paragraph">
<p>For example: <code>class AddImm</code>, which is what we get on a simple <code>add x1, x2, 0</code>, sets itself as an <code>IntAluOp</code> on the constructor as expected:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    AddImm::AddImm(ExtMachInst machInst,
                                          IntRegIndex _dest,
                                          IntRegIndex _op1,
                                          uint32_t _imm,
                                          bool _rotC)
        : DataImmOp("add", machInst, IntAluOp,
                         _dest, _op1, _imm, _rotC)</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-derivo3cpu-default-functional-units"><a class="anchor" href="#gem5-derivo3cpu-default-functional-units"></a><a class="link" href="#gem5-derivo3cpu-default-functional-units">24.22.7.2. gem5 DerivO3CPU default functional units</a></h5>
<div class="paragraph">
<p>On gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772, after running:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run   \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
  --trace-insts-stdout \
  -- \
  --cpu-type Derivo3CPU \
  --caches</pre>
</div>
</div>
<div class="paragraph">
<p>we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu]
type=DerivO3CPU
children=branchPred dcache dtb fuPool icache interrupts isa itb power_state tracer workload</pre>
</div>
</div>
<div class="paragraph">
<p>and following <code>fuPool</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.fuPool]
type=FUPool
children=FUList0 FUList1 FUList2 FUList3 FUList4 FUList5 FUList6 FUList7 FUList8 FUList9</pre>
</div>
</div>
<div class="paragraph">
<p>so for example <code>FUList0</code> is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.fuPool.FUList0]
type=FUDesc
children=opList
count=6
eventq_index=0
opList=system.cpu.fuPool.FUList0.opList

[system.cpu.fuPool.FUList0.opList]
type=OpDesc
eventq_index=0
opClass=IntAlu
opLat=1
pipelined=true</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>FUList1</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>[system.cpu.fuPool.FUList1.opList0]
type=OpDesc
eventq_index=0
opClass=IntMult
opLat=3
pipelined=true

[system.cpu.fuPool.FUList1.opList1]
type=OpDesc
eventq_index=0
opClass=IntDiv
opLat=20
pipelined=false</pre>
</div>
</div>
<div class="paragraph">
<p>So summarizing all units we have:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>0, 1: <code>IntAlu</code> with <code>opLat=3</code></p>
</li>
<li>
<p>2: <code>IntMult</code> with <code>opLat=3</code> and <code>IntDiv</code> with <code>opLat=20</code></p>
</li>
<li>
<p>3: <code>FloatAdd</code>, <code>FloatCmp</code>, <code>FloatCvt</code> with <code>opLat=2</code></p>
</li>
<li>
<p>TODO lazy to finish the list :-)</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-code-generation"><a class="anchor" href="#gem5-code-generation"></a><a class="link" href="#gem5-code-generation">24.22.8. gem5 code generation</a></h4>
<div class="paragraph">
<p>gem5 uses a ton of code generation, which makes the project horrendous:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>lots of magic happen on top of pybind11, which is already magic, to more automatically glue the C++ and Python worlds: <a href="#gem5-python-c-interaction">gem5 Python C++ interaction</a></p>
</li>
<li>
<p>.isa code which describes most of the instructions: <a href="#gem5-instruction-definitions">gem5 instruction definitions</a></p>
</li>
<li>
<p><a href="#gem5-ruby-build">Ruby</a> for memory systems</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To find files that are not symlinks use <a href="https://stackoverflow.com/questions/16303449/how-to-find-files-excluding-symbolic-links" class="bare">https://stackoverflow.com/questions/16303449/how-to-find-files-excluding-symbolic-links</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>find build -type f</pre>
</div>
</div>
<div class="paragraph">
<p>To find the definition of generated code, do a:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep -I -r build/ 'code of interest'</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>-I</code>: ignore binray file matches on built objects</p>
</li>
<li>
<p><code>-r</code>: ignore symlinks due to <a href="#why-are-all-c-symlinked-into-the-gem5-build-dir">Why are all C++ symlinked into the gem5 build dir?</a> as explained at <a href="https://stackoverflow.com/questions/21738574/how-do-you-exclude-symlinks-in-a-grep" class="bare">https://stackoverflow.com/questions/21738574/how-do-you-exclude-symlinks-in-a-grep</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The code generation exists partly to support insanely generic cross ISA instructions mapping to one compute model, where it might be reasonable.</p>
</div>
<div class="paragraph">
<p>But it has been widely overused to insanity. It likely also exists partly because when the project started in 2003 C++ compilers weren&#8217;t that good, so you couldn&#8217;t rely on features like templates that much.</p>
</div>
<div class="sect4">
<h5 id="gem5-the-isa"><a class="anchor" href="#gem5-the-isa"></a><a class="link" href="#gem5-the-isa">24.22.8.1. gem5 THE_ISA</a></h5>
<div class="paragraph">
<p>Generated code at: <code>build/&lt;ISA&gt;/config/the_isa.hh</code> which e.g. for ARM contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#ifndef __CONFIG_THE_ISA_HH__
#define __CONFIG_THE_ISA_HH__

#define ARM_ISA 1
#define MIPS_ISA 2
#define NULL_ISA 3
#define POWER_ISA 4
#define RISCV_ISA 5
#define SPARC_ISA 6
#define X86_ISA 7

enum class Arch {
  ArmISA = ARM_ISA,
  MipsISA = MIPS_ISA,
  NullISA = NULL_ISA,
  PowerISA = POWER_ISA,
  RiscvISA = RISCV_ISA,
  SparcISA = SPARC_ISA,
  X86ISA = X86_ISA
};

#define THE_ISA ARM_ISA
#define TheISA ArmISA
#define THE_ISA_STR "arm"

#endif // __CONFIG_THE_ISA_HH__</pre>
</div>
</div>
<div class="paragraph">
<p>Generation code: <code>src/SConscript</code> at <code>def makeTheISA</code>.</p>
</div>
<div class="paragraph">
<p>Tested on gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://www.mail-archive.com/gem5-users@gem5.org/msg16989.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg16989.html</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-build-system"><a class="anchor" href="#gem5-build-system"></a><a class="link" href="#gem5-build-system">24.22.9. gem5 build system</a></h4>
<div class="sect4">
<h5 id="m5-override-py-source"><a class="anchor" href="#m5-override-py-source"></a><a class="link" href="#m5-override-py-source">24.22.9.1. M5_OVERRIDE_PY_SOURCE</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5" class="bare">https://stackoverflow.com/questions/52312070/how-to-modify-a-file-under-src-python-and-run-it-without-rebuilding-in-gem5</a></p>
</div>
<div class="paragraph">
<p>Running gem5 with the <code>M5_OVERRIDE_PY_SOURCE=true</code> environment variable allows you to modify a file under src/python and run it without rebuilding in gem5?</p>
</div>
<div class="paragraph">
<p>We set this environment variable by default in our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run">run</a> script.</p>
</div>
<div class="paragraph">
<p>How <code>M5_OVERRID_PY_SOURCE</code> works is shown at: <a href="#gem5-m5-objects-module">gem5 <code>m5.objects</code> module</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-build-broken-on-recent-compiler-version"><a class="anchor" href="#gem5-build-broken-on-recent-compiler-version"></a><a class="link" href="#gem5-build-broken-on-recent-compiler-version">24.22.9.2. gem5 build broken on recent compiler version</a></h5>
<div class="paragraph">
<p>gem5 moves a bit slowly, and if your host compiler is very new, the gem5 build might be broken for it, e.g. this was the case for Ubuntu 19.10 with GCC 9 and gem5 62d75e7105fe172eb906d4f80f360ff8591d4178 from Dec 2019.</p>
</div>
<div class="paragraph">
<p>This happens mostly because GCC keeps getting more strict with warnings and gem5 uses <code>-Werror</code>.</p>
</div>
<div class="paragraph">
<p>The specific problem mentioned above was later fixed, but if it ever happens again, you can work around it by either by or by disabling <code>-Werror</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 -- CCFLAGS=-Wno-error</pre>
</div>
</div>
<div class="paragraph">
<p>or by  <a href="https://askubuntu.com/questions/466651/how-do-i-use-the-latest-gcc-on-ubuntu/1163021#1163021">installing an older compiler</a> and using it with something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 -- CC=gcc-8 CXX=g++-8</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gem5-polymorphic-isa-includes"><a class="anchor" href="#gem5-polymorphic-isa-includes"></a><a class="link" href="#gem5-polymorphic-isa-includes">24.22.9.3. gem5 polymorphic ISA includes</a></h5>
<div class="paragraph">
<p>E.g. <code>src/cpu/decode_cache.hh</code> includes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#include "arch/isa_traits.hh"</pre>
</div>
</div>
<div class="paragraph">
<p>which in turn is meant to refer to files of form:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>src/arch/&lt;isa&gt;/isa_traits.hh</pre>
</div>
</div>
<div class="paragraph">
<p>What happens is that the build system creates a file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/arch/isa_traits.hh</pre>
</div>
</div>
<div class="paragraph">
<p>which contains just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#include "arch/arm/isa_traits.hh"</pre>
</div>
</div>
<div class="paragraph">
<p>and puts that in the <code>-I</code> include path during build.</p>
</div>
<div class="paragraph">
<p>It appears to be possible to deal with it using preprocessor macros, but it is ugly: <a href="https://stackoverflow.com/questions/3178946/using-define-to-include-another-file-in-c-c/3179218#3179218" class="bare">https://stackoverflow.com/questions/3178946/using-define-to-include-another-file-in-c-c/3179218#3179218</a></p>
</div>
<div class="paragraph">
<p>In addition to the header polymorphism, gem5 also namespaces classes with <code>TheISA::</code>, e.g. in <code>src/cpu/decode_cache.hh</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Value items[TheISA::PageBytes];</pre>
</div>
</div>
<div class="paragraph">
<p>which is defined at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/config/the_isa.hh</pre>
</div>
</div>
<div class="paragraph">
<p>as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#define TheISA ArmISA</pre>
</div>
</div>
<div class="paragraph">
<p>and forces already <code>arm/</code> specific headers to define their symbols under:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>namespace ArmISA</pre>
</div>
</div>
<div class="paragraph">
<p>so I don&#8217;t see the point of this pattern, why not just us <code>PageBytes</code> directly? Looks like a documentation mechanism to indicate that a certain symbol is ISA specific.</p>
</div>
<div class="paragraph">
<p>Tested in gem5 2a242c5f59a54bc6b8953f82486f7e6fe0aa9b3d.</p>
</div>
</div>
<div class="sect4">
<h5 id="why-are-all-c-symlinked-into-the-gem5-build-dir"><a class="anchor" href="#why-are-all-c-symlinked-into-the-gem5-build-dir"></a><a class="link" href="#why-are-all-c-symlinked-into-the-gem5-build-dir">24.22.9.4. Why are all C++ symlinked into the gem5 build dir?</a></h5>
<div class="paragraph">
<p>Upstream request: <a href="https://gem5.atlassian.net/browse/GEM5-469" class="bare">https://gem5.atlassian.net/browse/GEM5-469</a></p>
</div>
<div class="paragraph">
<p>Some scons madness.</p>
</div>
<div class="paragraph">
<p><a href="https://scons.org/doc/2.4.1/HTML/scons-user.html#idp1378838508" class="bare">https://scons.org/doc/2.4.1/HTML/scons-user.html#idp1378838508</a> generates hard links by default.</p>
</div>
<div class="paragraph">
<p>Then the a5bc2291391b0497fdc60fdc960e07bcecebfb8f SConstruct use symlinks in a futile attempt to make things better for editors or build systems from the past century.</p>
</div>
<div class="paragraph">
<p>It was not possible to disable the symlinks automatically for the entire project when I last asked: <a href="https://stackoverflow.com/questions/53656787/how-to-set-disable-duplicate-0-for-all-scons-build-variants-without-repeating-th" class="bare">https://stackoverflow.com/questions/53656787/how-to-set-disable-duplicate-0-for-all-scons-build-variants-without-repeating-th</a></p>
</div>
<div class="paragraph">
<p>The horrendous downsides of this are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>it is basically impossible to setup an IDE properly with gem5: <a href="#gem5-eclipse-configuration">gem5 Eclipse configuration</a></p>
</li>
<li>
<p>It is likely preventing <a href="#ccache">ccache</a> hits when building to different output paths, because it makes the <code>-I</code> includes point to different paths. This is especially important for <a href="#gem5-ruby-build">gem5 Ruby build</a>, which could have the exact same source files as the non-Ruby builds: <a href="https://stackoverflow.com/questions/60340271/can-ccache-handle-symlinks-to-the-same-input-source-file-as-hits" class="bare">https://stackoverflow.com/questions/60340271/can-ccache-handle-symlinks-to-the-same-input-source-file-as-hits</a></p>
</li>
<li>
<p>when <a href="#debug-the-emulator">debugging the emulator</a>, it shows you directories inside the build directory rather than in the source tree</p>
</li>
<li>
<p>it is harder to separate which files are <a href="#gem5-code-generation">generated</a> and which are in-tree when grepping for code generated definitions</p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="gensim"><a class="anchor" href="#gensim"></a><a class="link" href="#gensim">25. Gensim</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://gensim.org" class="bare">https://gensim.org</a></p>
</div>
<div class="paragraph">
<p>Source at: <a href="https://github.com/gensim-project/gensim" class="bare">https://github.com/gensim-project/gensim</a> previously at: <a href="https://bitbucket.org/gensim/gensim" class="bare">https://bitbucket.org/gensim/gensim</a></p>
</div>
<div class="paragraph">
<p>MIT licensed <a href="#binary-translation">Binary translation</a> simulator, so a bit like an MIT <a href="#qemu">QEMU</a>.</p>
</div>
<div class="paragraph">
<p>Video showing it boot Linux fast: <a href="https://www.youtube.com/watch?v=aZXx17oYumc" class="bare">https://www.youtube.com/watch?v=aZXx17oYumc</a></p>
</div>
<div class="paragraph">
<p>Its name is unfortunately completely and totally overshadowed by an unrelated software with the sane name: <a href="https://radimrehurek.com/gensim/" class="bare">https://radimrehurek.com/gensim/</a></p>
</div>
<div class="paragraph">
<p>TODO: advantages over QEMU. Like the name implies, they seem to have a nice ISA description language. From quick internals look, seems to generate LLVM intermediate language, which sound good.</p>
</div>
<div class="paragraph">
<p>Build on Ubuntu 20.04:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git submodule update --init submodules/gensim
sudo apt install libantlr3c-dev
cd submodule/gensim
make</pre>
</div>
</div>
<div class="paragraph">
<p>First fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>arm-none-eabi-gcc: error: unrecognized -march target: armv5</pre>
</div>
</div>
<div class="paragraph">
<p>Let&#8217;s try just armv8, who cares about arvm5!!!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir build
cd build
cmake -DTESTING_ENABLED=FALSE -DCMAKE_BUILD_TYPE=DEBUGOPT ..
make -j`nproc` model-armv8</pre>
</div>
</div>
<div class="paragraph">
<p>Now fails as mentioned at <a href="https://bitbucket.org/gensim/gensim/issues/34/build-fails-with-unrecognised-intrinsic" class="bare">https://bitbucket.org/gensim/gensim/issues/34/build-fails-with-unrecognised-intrinsic</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>terminate called after throwing an instance of 'std::logic_error'
  what():  Unrecognised intrinsic: __builtin_abs64
Aborted (core dumped)</pre>
</div>
</div>
<div class="paragraph">
<p>Get the failing command with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>make VERBOSE=1 model-armv8</pre>
</div>
</div>
<div class="paragraph">
<p>and we see some code generation step:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd /home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/models/armv8 &amp;&amp; \
  /home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/build/dist/bin/gensim \
  -a /home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/models/armv8/aarch64.ac \
  -s module,arch,decode,disasm,ee_interp,ee_blockjit,jumpinfo,function,makefile \
  -o decode.GenerateDotGraph=1,makefile.libtrace_path=/home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/support/libtrace/inc,makefile.archsim_path=/home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/archsim/inc,makefile.llvm_path=,makefile.Optimise=2,makefile.Debug=1 \
  -t /home/ciro/bak/git/linux-kernel-module-cheat/submodules/gensim/build/models/armv8/output-aarch64/</pre>
</div>
</div>
<div class="paragraph">
<p>We can see an inclusion path:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gensim/models/armv8/aarch64.ac
		ac_isa("isa.ac");
gensim/models/armv8/isa.ac
		ac_execute("execute.simd");</pre>
</div>
</div>
<div class="paragraph">
<p>and where <code>gensim/models/armv8/isa.ac</code> contains <code>__builtin_abs64</code> usages.</p>
</div>
<div class="paragraph">
<p>Rebuilding with <code>-DCMAKE_BUILD_TYPE=DEBUG</code> + GDB on <code>gensim</code> shows that the error comes from a call to <code>gci.GenerateExecuteBodyFor(body_str, *action);</code>, so it looks like there are some missing cases in <code>gensim/src/generators/GenCInterpreter/InterpreterNodeWalker.cpp</code> function <code>SSAIntrinsicStatementWalker::EmitFixedCode</code>, e.g. there should be one for <code>__builtin_abs64</code>.</p>
</div>
<div class="paragraph">
<p>This is completely broken academic code! They must be using an off-tree of part of the tool and forgot to commit.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="buildroot"><a class="anchor" href="#buildroot"></a><a class="link" href="#buildroot">26. Buildroot</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="introduction-to-buildroot"><a class="anchor" href="#introduction-to-buildroot"></a><a class="link" href="#introduction-to-buildroot">26.1. Introduction to Buildroot</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Buildroot">Buildroot</a> is a set of Make scripts that download and compile from source compatible versions of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>GCC</p>
</li>
<li>
<p>Linux kernel</p>
</li>
<li>
<p>C standard library: Buildroot supports several implementations, see: <a href="#libc-choice">Section 26.10, &#8220;libc choice&#8221;</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/BusyBox">BusyBox</a>: provides the shell and basic command line utilities</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It therefore produces a pristine, blob-less, debuggable setup, where all moving parts are configured to work perfectly together.</p>
</div>
<div class="paragraph">
<p>Perhaps the awesomeness of Buildroot only sinks in once you notice that all it takes is 4 commands as explained at <a href="#buildroot-hello-world">Section 26.11, &#8220;Buildroot hello world&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The downsides of Buildroot are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the first build takes a while compared to downloading prebuilts, but it is well worth it</p>
</li>
<li>
<p>the selection of software packages is relatively limited if compared to Debian.</p>
<div class="paragraph">
<p>In theory, any software can be packaged, and the Buildroot side is easy.</p>
</div>
<div class="paragraph">
<p>The hard part is dealing with crappy third party build systems and huge dependency chains.</p>
</div>
</li>
<li>
<p>it is written in Make and Bash rather than Python like LKMC</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This repo basically wraps around that, and tries to make everything even more awesome for kernel developers by adding the capability of seamlessly running the stuff you&#8217;ve built on emulators usually via <code>./run</code>.</p>
</div>
<div class="paragraph">
<p>This runnable part of selecting the command line options for different emulators and setups is to a large extent what <a href="https://en.wikipedia.org/wiki/Libvirt">Libvirt</a> does. But we feel that having both build and run on the same repository is the key.</p>
</div>
<div class="paragraph">
<p>As this repo develops however, we&#8217;ve started taking some of the build out of Buildroot, e.g. notably the <a href="#buildroot-vanilla-kernel">Linux kernel</a> to have more build flexibility and faster build startup times.</p>
</div>
<div class="paragraph">
<p>Therefore, more and more, this repo wants to take over everything that Buildroot does, and one day completely replace it to achieve emulation Nirvana, see e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/116" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/116</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/117" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/117</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="custom-buildroot-configs"><a class="anchor" href="#custom-buildroot-configs"></a><a class="link" href="#custom-buildroot-configs">26.2. Custom Buildroot configs</a></h3>
<div class="paragraph">
<p>We provide the following mechanisms:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>./build-buildroot --config-fragment data/br2</code>: append the Buildroot configuration file <code>data/br2</code> to a single build. Must be passed every time you run <code>./build</code>. The format is the same as <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_config/default">buildroot_config/default</a>.</p>
</li>
<li>
<p><code>./build-buildroot --config 'BR2_SOME_OPTION="myval"'</code>: append a single option to a single build.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For example, if you decide to <a href="#enable-buildroot-compiler-optimizations">Enable Buildroot compiler optimizations</a> after an initial build is finished, you must <a href="#clean-the-build">Clean the build</a> and rebuild:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --config 'BR2_OPTIMIZE_3=y' \
  --config 'BR2_PACKAGE_SAMPLE_PACKAGE=y' \
  --
  sample_package-dirclean \
  sample_package-reconfigure \
;</pre>
</div>
</div>
<div class="paragraph">
<p>as explained at: <a href="https://buildroot.org/downloads/manual/manual.html#rebuild-pkg" class="bare">https://buildroot.org/downloads/manual/manual.html#rebuild-pkg</a></p>
</div>
<div class="paragraph">
<p>The clean is necessary because the source files didn&#8217;t change, so <code>make</code> would just check the timestamps and not build anything.</p>
</div>
<div class="paragraph">
<p>You will then likely want to make those more permanent as explained at: <a href="#default-command-line-arguments">Section 38.4, &#8220;Default command line arguments&#8221;</a>.</p>
</div>
<div class="sect3">
<h4 id="enable-buildroot-compiler-optimizations"><a class="anchor" href="#enable-buildroot-compiler-optimizations"></a><a class="link" href="#enable-buildroot-compiler-optimizations">26.2.1. Enable Buildroot compiler optimizations</a></h4>
<div class="paragraph">
<p>If you are benchmarking compiled programs instead of hand written assembly, remember that we configure Buildroot to disable optimizations by default with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BR2_OPTIMIZE_0=y</pre>
</div>
</div>
<div class="paragraph">
<p>to improve the debugging experience.</p>
</div>
<div class="paragraph">
<p>You will likely want to change that to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BR2_OPTIMIZE_3=y</pre>
</div>
</div>
<div class="paragraph">
<p>Our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/sample_package">buildroot_packages/sample_package</a> package correctly forwards the Buildroot options to the build with <code>$(TARGET_CONFIGURE_OPTS)</code>, so you don&#8217;t have to do any extra work.</p>
</div>
<div class="paragraph">
<p>Don&#8217;t forget to do that if you are <a href="#add-new-buildroot-packages">adding a new package</a> with your own build system.</p>
</div>
<div class="paragraph">
<p>Then, you have two choices:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>if you already have a full <code>-O0</code> build, you can choose to rebuild just your package of interest to save some time as described at: <a href="#custom-buildroot-configs">Section 26.2, &#8220;Custom Buildroot configs&#8221;</a></p>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --config 'BR2_OPTIMIZE_3=y' \
  --config 'BR2_PACKAGE_SAMPLE_PACKAGE=y' \
  -- \
  sample_package-dirclean \
  sample_package-reconfigure \
;</pre>
</div>
</div>
<div class="paragraph">
<p>However, this approach might not be representative since calls to an unoptimized libc and other libraries will have a negative performance impact.</p>
</div>
<div class="paragraph">
<p>Maybe you can get away with rebuilding libc, but I&#8217;m not sure that it will work properly.</p>
</div>
<div class="paragraph">
<p>Kernel-wise it should be fine though as mentioned at: <a href="#kernel-o0">Section 3.1.2, &#8220;Disable kernel compiler optimizations&#8221;</a></p>
</div>
</li>
<li>
<p><a href="#clean-the-build">clean the build</a> and rebuild from scratch:</p>
<div class="literalblock">
<div class="content">
<pre>mv out out~
./build-buildroot --config 'BR2_OPTIMIZE_3=y'</pre>
</div>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="find-buildroot-options-with-make-menuconfig"><a class="anchor" href="#find-buildroot-options-with-make-menuconfig"></a><a class="link" href="#find-buildroot-options-with-make-menuconfig">26.3. Find Buildroot options with make menuconfig</a></h3>
<div class="paragraph">
<p><code>make menuconfig</code> is a convenient way to find Buildroot configurations:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd "$(./getvar buildroot_build_dir)"
make menuconfig</pre>
</div>
</div>
<div class="paragraph">
<p>Hit <code>/</code> and search for the settings.</p>
</div>
<div class="paragraph">
<p>Save and quit.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>diff -u .config.olg .config</pre>
</div>
</div>
<div class="paragraph">
<p>Then copy and paste the diff additions to <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_config/default">buildroot_config/default</a> to make them permanent.</p>
</div>
</div>
<div class="sect2">
<h3 id="change-user"><a class="anchor" href="#change-user"></a><a class="link" href="#change-user">26.4. Change user</a></h3>
<div class="paragraph">
<p>At startup, we login automatically as the <code>root</code> user.</p>
</div>
<div class="paragraph">
<p>If you want to switch to another user to test some permissions, we have already created an <code>user0</code> user through the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/user_table">user_table</a> file, and you can just login as that user with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>login user0</pre>
</div>
</div>
<div class="paragraph">
<p>and password:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>a</pre>
</div>
</div>
<div class="paragraph">
<p>Then test that the user changed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>id</pre>
</div>
</div>
<div class="paragraph">
<p>which gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>uid=1000(user0) gid=1000(user0) groups=1000(user0)</pre>
</div>
</div>
<div class="sect3">
<h4 id="login-as-a-non-root-user-without-password"><a class="anchor" href="#login-as-a-non-root-user-without-password"></a><a class="link" href="#login-as-a-non-root-user-without-password">26.4.1. Login as a non-root user without password</a></h4>
<div class="paragraph">
<p>Replace on <code>inittab</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>::respawn:-/bin/sh</pre>
</div>
</div>
<div class="paragraph">
<p>with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>::respawn:-/bin/login -f user0</pre>
</div>
</div>
<div class="paragraph">
<p><code>-f</code> forces login without asking for the password.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="add-new-files-to-the-buildroot-image"><a class="anchor" href="#add-new-files-to-the-buildroot-image"></a><a class="link" href="#add-new-files-to-the-buildroot-image">26.5. Add new files to the Buildroot image</a></h3>
<div class="paragraph">
<p>These are your options:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>create a Buildroot package: <a href="#add-new-buildroot-packages">Add new Buildroot packages</a></p>
<div class="paragraph">
<p>This is the most general option, but the most laborious. No big deal if you copy our template however as shown in that section.</p>
</div>
<div class="paragraph">
<p>Handles any type of cross compilation, including multiple input sources.</p>
</div>
</li>
<li>
<p>drop your files directly in <a href="#rootfs-overlay">rootfs_overlay</a> and follow instructions from that section.</p>
<div class="paragraph">
<p>Files in that directory are directly copied to the image, so this is the best option for files that don&#8217;t need to be compiled such as <a href="#interpreted-languages">Interpreted languages</a>.</p>
</div>
<div class="paragraph">
<p>You could also use this method to inject compiled binaries into the image for quick-and-dirty testing.</p>
</div>
<div class="paragraph">
<p>But it will be much more likely to work if you use our cross compiler with <a href="#run-toolchain">run-toolchain</a> or <a href="#getvar">getvar</a>.</p>
</div>
<div class="paragraph">
<p>If you can&#8217;t do that, at the very least make it statically with <code>-static</code> compiled to remove the possibility of binary mismatch with our dynamic glibc.</p>
</div>
<div class="paragraph">
<p>But things can still break if your random glibc is configured to work with a newer Linux kernel than ours.</p>
</div>
<div class="paragraph">
<p>It often just works even if they are not perfectly matched however, partly because the Linux kernel is highly <a href="#update-the-linux-kernel">backwards compatible</a></p>
</div>
</li>
<li>
<p>fork this repo and add new files to <a href="#userland-content">userland/</a> or <a href="#kernel-modules">kernel_modules/</a></p>
<div class="paragraph">
<p>To add a simple executable that compiles from a single source file, like the dozens of examples that we have, you could just go this route.</p>
</div>
<div class="paragraph">
<p>This mechanisms bypasses having to create/modify Buildroot packages, and is very simple when you have a single input single output executable.</p>
</div>
</li>
<li>
<p><a href="#9p">9P</a>. OK, this is not really adding to the image, but it is the most convenient way to quickly modify a binary on the host, cross compile, and test it out without rebooting.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Related threads:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/22" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/22</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/50" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/50</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="add-new-buildroot-packages"><a class="anchor" href="#add-new-buildroot-packages"></a><a class="link" href="#add-new-buildroot-packages">26.5.1. Add new Buildroot packages</a></h4>
<div class="paragraph">
<p>First, see if you can&#8217;t get away without actually adding a new package, for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>if you have a standalone C file with no dependencies besides the C standard library to be compiled with GCC, just add a new file under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/sample_package">buildroot_packages/sample_package</a> and you are done</p>
</li>
<li>
<p>if you have a dependency on a library, first check if Buildroot doesn&#8217;t have a package for it already with <code>ls buildroot/package</code>. If yes, just enable that package as explained at: <a href="#custom-buildroot-configs">Section 26.2, &#8220;Custom Buildroot configs&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If none of those methods are flexible enough for you, you can just fork or hack up <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/sample_package">buildroot_packages/sample_package</a> the sample package to do what you want.</p>
</div>
<div class="paragraph">
<p>For how to use that package, see: <a href="#buildroot-packages-directory">Section 38.15.2, &#8220;buildroot_packages directory&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Then iterate trying to do what you want and reading the manual until it works: <a href="https://buildroot.org/downloads/manual/manual.html" class="bare">https://buildroot.org/downloads/manual/manual.html</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="remove-buildroot-packages"><a class="anchor" href="#remove-buildroot-packages"></a><a class="link" href="#remove-buildroot-packages">26.6. Remove Buildroot packages</a></h3>
<div class="paragraph">
<p>Once you&#8217;ve built a package in to the image, there is no easy way to remove it.</p>
</div>
<div class="paragraph">
<p>Documented at: <a href="https://github.com/buildroot/buildroot/blob/2017.08/docs/manual/rebuilding-packages.txt#L90" class="bare">https://github.com/buildroot/buildroot/blob/2017.08/docs/manual/rebuilding-packages.txt#L90</a></p>
</div>
<div class="paragraph">
<p>Also mentioned at: <a href="https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot" class="bare">https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot</a></p>
</div>
</div>
<div class="sect2">
<h3 id="br2-target-rootfs-ext2-size"><a class="anchor" href="#br2-target-rootfs-ext2-size"></a><a class="link" href="#br2-target-rootfs-ext2-size">26.7. BR2_TARGET_ROOTFS_EXT2_SIZE</a></h3>
<div class="paragraph">
<p>When adding new large package to the Buildroot root filesystem, it may fail with the message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Maybe you need to increase the filesystem size (BR2_TARGET_ROOTFS_EXT2_SIZE)</pre>
</div>
</div>
<div class="paragraph">
<p>The solution is to simply add:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_TARGET_ROOTFS_EXT2_SIZE="512M"'</pre>
</div>
</div>
<div class="paragraph">
<p>where 512Mb is "large enough".</p>
</div>
<div class="paragraph">
<p>Note that dots cannot be used as in <code>1.5G</code>, so just use Megs as in <code>1500M</code> instead.</p>
</div>
<div class="paragraph">
<p>Unfortunately, TODO we don&#8217;t have a perfect way to find the right value for <code>BR2_TARGET_ROOTFS_EXT2_SIZE</code>. One good heuristic is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>du -hsx "$(./getvar --arch arm buildroot_target_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>Some promising ways to overcome this problem include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#squashfs">SquashFS</a>
TODO benchmark: would gem5 suffer a considerable disk read performance hit due to decompressing SquashFS?</p>
</li>
<li>
<p>libguestfs: <a href="https://serverfault.com/questions/246835/convert-directory-to-qemu-kvm-virtual-disk-image/916697#916697" class="bare">https://serverfault.com/questions/246835/convert-directory-to-qemu-kvm-virtual-disk-image/916697#916697</a>, in particular <a href="http://libguestfs.org/guestfish.1.html#vfs-minimum-size"><code>vfs-minimum-size</code></a></p>
</li>
<li>
<p>use methods described at: <a href="#gem5-restore-new-script">Section 24.6.3, &#8220;gem5 checkpoint restore and run a different script&#8221;</a> instead of putting builds on the root filesystem</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/49211241/is-there-a-way-to-automatically-detect-the-minimum-required-br2-target-rootfs-ex" class="bare">https://stackoverflow.com/questions/49211241/is-there-a-way-to-automatically-detect-the-minimum-required-br2-target-rootfs-ex</a></p>
</div>
<div class="sect3">
<h4 id="squashfs"><a class="anchor" href="#squashfs"></a><a class="link" href="#squashfs">26.7.1. SquashFS</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/SquashFS">SquashFS</a> creation with <code>mksquashfs</code> does not take fixed sizes, and I have successfully booted from it, but it is readonly, which is unacceptable.</p>
</div>
<div class="paragraph">
<p>But then we could mount <a href="https://wiki.debian.org/ramfs">ramfs</a> on top of it with <a href="#overlayfs">OverlayFS</a> to make it writable, but my attempts failed exactly as mentioned at <a href="#overlayfs">OverlayFS</a>.</p>
</div>
<div class="paragraph">
<p>This is the exact unanswered question: <a href="https://unix.stackexchange.com/questions/343484/mounting-squashfs-image-with-read-write-overlay-for-rootfs" class="bare">https://unix.stackexchange.com/questions/343484/mounting-squashfs-image-with-read-write-overlay-for-rootfs</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="rpath"><a class="anchor" href="#rpath"></a><a class="link" href="#rpath">26.8. Buildroot rebuild is slow when the root filesystem is large</a></h3>
<div class="paragraph">
<p>Buildroot is not designed for large root filesystem images, and the rebuild becomes very slow when we add a large package to it.</p>
</div>
<div class="paragraph">
<p>This is due mainly to the <code>pkg-generic</code> <code>GLOBAL_INSTRUMENTATION_HOOKS</code> sanitation which go over the entire tree doing complex operations&#8230;&#8203; I no like, in particular <code>check_bin_arch</code> and <code>check_host_rpath</code></p>
</div>
<div class="paragraph">
<p>We have applied <a href="https://github.com/cirosantilli/buildroot/commit/983fe7910a73923a4331e7d576a1e93841d53812">983fe7910a73923a4331e7d576a1e93841d53812</a> to out Buildroot fork which removes part of the pain by not running:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&gt;&gt;&gt;   Sanitizing RPATH in target tree</pre>
</div>
</div>
<div class="paragraph">
<p>which contributed to a large part of the slowness.</p>
</div>
<div class="paragraph">
<p>Test how Buildroot deals with many files with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --config 'BR2_PACKAGE_LKMC_MANY_FILES=y' \
  -- \
  lkmc_many_files-reconfigure \
  |&amp; \
  ts -i '%.s' \
;
./build-buildroot |&amp; ts -i '%.s'</pre>
</div>
</div>
<div class="paragraph">
<p>and notice how the second build, which does not rebuilt the package at all, still gets stuck in the <code>RPATH</code> check forever without our Buildroot patch.</p>
</div>
</div>
<div class="sect2">
<h3 id="report-upstream-bugs"><a class="anchor" href="#report-upstream-bugs"></a><a class="link" href="#report-upstream-bugs">26.9. Report upstream bugs</a></h3>
<div class="paragraph">
<p>When asking for help on upstream repositories outside of this repository, you will need to provide the commands that you are running in detail without referencing our scripts.</p>
</div>
<div class="paragraph">
<p>For example, QEMU developers will only want to see the final QEMU command that you are running.</p>
</div>
<div class="paragraph">
<p>For the configure and build, search for the <code>Building</code> and <code>Configuring</code> parts of the build log, then try to strip down all Buildroot related paths, to keep only options that seem to matter.</p>
</div>
<div class="paragraph">
<p>We make that easy by building commands as strings, and then echoing them before evaling.</p>
</div>
<div class="paragraph">
<p>So for example when you run:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm</pre>
</div>
</div>
<div class="paragraph">
<p>the very first stdout output of that script is the actual QEMU command that is being run.</p>
</div>
<div class="paragraph">
<p>The command is also saved to a file for convenience:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat "$(./getvar --arch arm run_cmd_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>which you can manually modify and execute during your experiments later:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim "$(./getvar --arch arm run_cmd_file)"
./"$(./getvar --arch arm run_cmd_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>If you are not already on the master of the given component, you can do that neatly with <a href="#build-variants">Build variants</a>.</p>
</div>
<div class="paragraph">
<p>E.g., to check if a QEMU bug is still present on <code>master</code>, you can do as explained at <a href="#qemu-build-variants">QEMU build variants</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git -C "$(./getvar qemu_source_dir)" checkout master
./build-qemu --clean --qemu-build-id master
./build-qemu --qemu-build-id master
git -C "$(./getvar qemu_source_dir)" checkout -
./run --qemu-build-id master</pre>
</div>
</div>
<div class="paragraph">
<p>Then, you will also want to do a <a href="#bisection">Bisection</a> to pinpoint the exact commit to blame, and CC that developer.</p>
</div>
<div class="paragraph">
<p>Finally, give the images you used save upstream developers' time as shown at: <a href="#release-zip">Section 38.19.2, &#8220;release-zip&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>For Buildroot problems, you should wither provide the config you have:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar buildroot_config_file</pre>
</div>
</div>
<div class="paragraph">
<p>or try to reproduce with a minimal config, see: <a href="https://github.com/cirosantilli/buildroot/tree/in-tree-package-master" class="bare">https://github.com/cirosantilli/buildroot/tree/in-tree-package-master</a></p>
</div>
</div>
<div class="sect2">
<h3 id="libc-choice"><a class="anchor" href="#libc-choice"></a><a class="link" href="#libc-choice">26.10. libc choice</a></h3>
<div class="paragraph">
<p>Buildroot supports several libc implementations, including:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/GNU_C_Library">glibc</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/UClibc">uClibc</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We currently use glibc, which is selected by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>BR2_TOOLCHAIN_BUILDROOT_GLIBC=y</pre>
</div>
</div>
<div class="paragraph">
<p>Ideally we would like to use uClibc, as it is more minimal and easier to understand, but unfortunately there are some very few packages that use some weird glibc extension that uClibc hasn&#8217;t implemented yet, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#selinux">SELinux</a>. Trivial unmerged fix at: <a href="http://lists.busybox.net/pipermail/buildroot/2017-July/197793.html" class="bare">http://lists.busybox.net/pipermail/buildroot/2017-July/197793.html</a> just missing the uClibc option to expose <code>fts.h</code>&#8230;&#8203;</p>
</li>
<li>
<p><a href="#stress">stress</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The full list of unsupported packages can be found by grepping the Buildroot source:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git -C "$(./getvar buildroot_source_dir)" grep 'depends on BR2_TOOLCHAIN_USES_GLIBC'</pre>
</div>
</div>
<div class="paragraph">
<p>One "downside" of glibc is that it exercises much more kernel functionality on its more bloated pre-main init, which breaks user mode C hello worlds more often, see: <a href="#user-mode-simulation-with-glibc">Section 11.4, &#8220;User mode simulation with glibc&#8221;</a>. I quote "downside" because glibc is actually exposing emulator bugs which we should actually go and fix.</p>
</div>
</div>
<div class="sect2">
<h3 id="buildroot-hello-world"><a class="anchor" href="#buildroot-hello-world"></a><a class="link" href="#buildroot-hello-world">26.11. Buildroot hello world</a></h3>
<div class="paragraph">
<p>This repo doesn&#8217;t do much more other than setting a bunch of Buildroot configurations and building it.</p>
</div>
<div class="paragraph">
<p>The minimal work you have to do to get QEMU to boot Buildroot from scratch is tiny if, about 4 commands!</p>
</div>
<div class="paragraph">
<p>Here are some good working commands for several ISAs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64 <a href="https://unix.stackexchange.com/questions/44062/how-to-use-qemu-to-run-build-root-linux-images/543075#543075" class="bare">https://unix.stackexchange.com/questions/44062/how-to-use-qemu-to-run-build-root-linux-images/543075#543075</a></p>
<div class="ulist">
<ul>
<li>
<p>x86_64 X11 <a href="https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116" class="bare">https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116</a> Also mentioned at: <a href="#x11">Section 14.4, &#8220;X11 Buildroot&#8221;</a>.</p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64 <a href="https://stackoverflow.com/questions/47557262/how-to-download-the-torvalds-linux-kernel-master-recompile-it-and-boot-it-wi/49349237#49349237" class="bare">https://stackoverflow.com/questions/47557262/how-to-download-the-torvalds-linux-kernel-master-recompile-it-and-boot-it-wi/49349237#49349237</a></p>
<div class="ulist">
<ul>
<li>
<p>aarch64 U-Boot: <a href="https://stackoverflow.com/questions/58028789/how-to-boot-linux-aarch64-with-u-boot-with-buildroot-on-qemu" class="bare">https://stackoverflow.com/questions/58028789/how-to-boot-linux-aarch64-with-u-boot-with-buildroot-on-qemu</a> Also mentioned at: <a href="#u-boot">U-Boot</a>.</p>
</li>
</ul>
</div>
</li>
<li>
<p>arm <a href="https://stackoverflow.com/questions/38320066/how-to-run-linux-on-a-qemu-arm-versatile-machine/44099299#44099299" class="bare">https://stackoverflow.com/questions/38320066/how-to-run-linux-on-a-qemu-arm-versatile-machine/44099299#44099299</a></p>
</li>
<li>
<p>PPC <a href="https://stackoverflow.com/questions/48021127/build-powerpc-kernel-and-boot-powerpc-kernel-on-qemu/49349262#49349262" class="bare">https://stackoverflow.com/questions/48021127/build-powerpc-kernel-and-boot-powerpc-kernel-on-qemu/49349262#49349262</a> just work commands for PPC, comment on how to replace kernel</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These can come in handy if you want to debug something in Buildroot itself and possibly report an upstream bug.</p>
</div>
</div>
<div class="sect2">
<h3 id="update-the-buildroot-toolchain"><a class="anchor" href="#update-the-buildroot-toolchain"></a><a class="link" href="#update-the-buildroot-toolchain">26.12. Update the Buildroot toolchain</a></h3>
<div class="paragraph">
<p>Users of this repo will often want to update the compilation toolchain to the latest version to get fresh new features like new ISA instructions.</p>
</div>
<div class="paragraph">
<p>Because the toolchain is so complex and tightly knitted with the rest of the system, this is more of an art than a science.</p>
</div>
<div class="paragraph">
<p>However, it is not something to be feared, and you will get there without help in most cases.</p>
</div>
<div class="paragraph">
<p>In this section we cover the most common cases.</p>
</div>
<div class="sect3">
<h4 id="update-gcc-gcc-supported-by-buildroot"><a class="anchor" href="#update-gcc-gcc-supported-by-buildroot"></a><a class="link" href="#update-gcc-gcc-supported-by-buildroot">26.12.1. Update GCC: GCC supported by Buildroot</a></h4>
<div class="paragraph">
<p>This is of course the simplest case.</p>
</div>
<div class="paragraph">
<p>You can quickly determine all the GCC versions supported by Buildroot by looking at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>submodules/buildroot/package/gcc/Config.in.host</pre>
</div>
</div>
<div class="paragraph">
<p>For example, in Buildroot 2018.08, which was used at LKMC 5d10529c10ad8a4777b0bac1543320df0c89a1ce, the default toolchain was 7.3.0, and the latest supported one was 8.2.0.</p>
</div>
<div class="paragraph">
<p>To just upgrade the toolchain to 8.2.0, and rebuild some userland executables to later run them, we could do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd submodules/gcc
git fetch up
git checkout -b lkmc-gcc-8_2_0-release gcc-8_2_0-release
git am ../buildroot/package/gcc/8.2.0/*
cd ../..
./build-buildroot \
  --arch aarch64 \
  --buildroot-build-id gcc-8-2 \
  --config 'BR2_GCC_VERSION_8_X=y' \
  --config 'BR2_GCC_VERSION="8.2.0"' \
  --no-all \
  -- \
  toolchain \
;
./build-userland \
  --arch aarch64 \
  --buildroot-build-id gcc-8-2 \
  --out-rootfs-overlay-dir-prefix gcc-8-2 \
  --userland-build-id gcc-8-2 \
;
./build-buildroot --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>where the <code>toolchain</code> Buildroot target builds only Buildroot: <a href="https://stackoverflow.com/questions/44521150/buildroot-install-and-build-the-toolchain-only" class="bare">https://stackoverflow.com/questions/44521150/buildroot-install-and-build-the-toolchain-only</a></p>
</div>
<div class="paragraph">
<p>Note that this setup did not overwrite any of our default Buildroot due to careful namespacing with our <code>gcc-8-2</code> prefix!</p>
</div>
<div class="paragraph">
<p>Now you can either run the executables on <a href="#user-mode-simulation">User mode simulation</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --userland userland/c/hello.c --userland-build-id gcc-8-2</pre>
</div>
</div>
<div class="paragraph">
<p>or in full system with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --eval-after './gcc-8-2/c/hello.out'</pre>
</div>
</div>
<div class="paragraph">
<p>where the <code>gcc-8-2</code> prefix was added by <code>--out-rootfs-overlay-dir-prefix</code>.</p>
</div>
<div class="paragraph">
<p><a href="#arm-sve">ARM SVE</a> support was only added to GCC 8 and can be enabled with the flag: <code>-march=armv8.2-a+sve</code>.</p>
</div>
<div class="paragraph">
<p>We already even had a C SVE test in-tree, but it was disabled because the old toolchain does not support it.</p>
</div>
<div class="paragraph">
<p>So once the new GCC 8 toolchain was built, we can first enable that test by editing the <a href="#path-properties">path_properties.py</a> file to not skip C SVE tests anymore:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>                #os.path.splitext(self.path_components[-1])[1] == '.c' and self['arm_sve']</pre>
</div>
</div>
<div class="paragraph">
<p>and then rebuild run one of the experiments from <a href="#change-arm-sve-vector-length-in-emulators">Change ARM SVE vector length in emulators</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland \
  --arch aarch64 \
  --buildroot-build-id gcc-8-2 \
  --force-rebuild \
  --march=armv8.2-a+sve \
  --out-rootfs-overlay-dir-prefix gcc-8-2 \
  --static \
  --userland-build-id gcc-8-2 \
;
./run \
  --arch aarch64 \
  --userland userland/arch/aarch64/inline_asm/sve_addvl.c \
  --userland-build-id gcc-8-2 \
  --static \
  --gem5-worktree master \
  -- \
  --param 'system.cpu[:].isa[:].sve_vl_se = 4' \</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/87" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/87</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="update-gcc-gcc-not-supported-by-buildroot"><a class="anchor" href="#update-gcc-gcc-not-supported-by-buildroot"></a><a class="link" href="#update-gcc-gcc-not-supported-by-buildroot">26.12.2. Update GCC: GCC not supported by Buildroot</a></h4>
<div class="paragraph">
<p>Now it gets fun, but well, guess what, we will try to do the same as <a href="#update-gcc-gcc-supported-by-buildroot">Section 26.12.1, &#8220;Update GCC: GCC supported by Buildroot&#8221;</a> but:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>pick the Buildroot version that comes closest to the GCC you want</p>
</li>
<li>
<p>if any <code>git am</code> patches don&#8217;t apply, skip them</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Now, if things fail, you can try:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>if the GCC version is supported by a newer Buildroot version:</p>
<div class="ulist">
<ul>
<li>
<p>quick and dirty: see what they are doing differently there, and patch it in here</p>
</li>
<li>
<p>golden star: upgrade our default Buildroot, <a href="#test-this-repo">test it well</a>, and send a pull request!</p>
</li>
</ul>
</div>
</li>
<li>
<p>otherwise: OK, go and patch Buildroot, time to become a Buildroot dev</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Known setups:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Buildroot 2018.08:</p>
<div class="ulist">
<ul>
<li>
<p>GCC 8.3.0: OK</p>
</li>
<li>
<p>GCC 9.2.0: KO <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/97" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/97</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="buildroot-vanilla-kernel"><a class="anchor" href="#buildroot-vanilla-kernel"></a><a class="link" href="#buildroot-vanilla-kernel">26.13. Buildroot vanilla kernel</a></h3>
<div class="paragraph">
<p>By default, our build system uses <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-linux">build-linux</a>, and the Buildroot kernel build is disabled: <a href="https://stackoverflow.com/questions/52231793/can-buildroot-build-the-root-filesystem-without-building-the-linux-kernel" class="bare">https://stackoverflow.com/questions/52231793/can-buildroot-build-the-root-filesystem-without-building-the-linux-kernel</a></p>
</div>
<div class="paragraph">
<p>There are however some cases where we want that ability, e.g.: <a href="#kernel-modules-buildroot-package">kernel_modules buildroot package</a> and <a href="#benchmark-linux-kernel-boot">Benchmark Linux kernel boot</a>.</p>
</div>
<div class="paragraph">
<p>The build of the kernel can be enabled with the <code>--build-kernel</code> option of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-buildroot">build-buildroot</a>.</p>
</div>
<div class="paragraph">
<p>For example, to build the kernel and then boot it you could do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch aarch64 --build-linux
./run --arch aarch64 --linux-exec "$(./getvar --arch aarch64 TODO)/vmlinux"</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: fails on LKMC d53ffcff18aa26d24ea34b86fb80e4a5694378dch with "ERROR: No hash found for linux-4.19.16.tar.xz": <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/115" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/115</a></p>
</div>
<div class="paragraph">
<p>Note that this kernel is not configured at all by LKMC, and there is no support to do that currently: the Buildroot default kernel configs for a target are used unchanged, e.g. <code>make qemu_aarch64_virt_defconfig</code>,see also; <a href="#buildroot-kernel-config">About Buildroot&#8217;s kernel configs</a>.</p>
</div>
<div class="paragraph">
<p>Therefore, this kernel might be missing certain key capabilities, e.g. filesystem support required to boot.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="userland-content"><a class="anchor" href="#userland-content"></a><a class="link" href="#userland-content">27. Userland content</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>This section documents our test and educational userland content, such as <a href="#c">C</a>, <a href="#cpp">C++</a> and <a href="#posix">POSIX</a> examples, present mostly under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/">userland/</a>.</p>
</div>
<div class="paragraph">
<p>Getting started at: <a href="#userland-setup">Section 2.8, &#8220;Userland setup&#8221;</a></p>
</div>
<div class="paragraph">
<p>Userland assembly content is located at: <a href="#userland-assembly">Section 28, &#8220;Userland assembly&#8221;</a>. It was split from this section basically because we were hitting the HTML <code>h6</code> limit, stupid web :-)</p>
</div>
<div class="paragraph">
<p>This content makes up the bulk of the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/">userland/</a> directory.</p>
</div>
<div class="paragraph">
<p>The quickest way to run the arch agnostic examples, which comprise the majority of the examples, is natively as shown at: <a href="#userland-setup-getting-started-natively">Section 2.8.2.1, &#8220;Userland setup getting started natively&#8221;</a></p>
</div>
<div class="paragraph">
<p>This section was originally moved in here from: <a href="https://github.com/cirosantilli/cpp-cheat" class="bare">https://github.com/cirosantilli/cpp-cheat</a></p>
</div>
<div class="sect2">
<h3 id="build-userland"><a class="anchor" href="#build-userland"></a><a class="link" href="#build-userland">27.1. build-userland</a></h3>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-userland">build-userland</a></p>
</div>
<div class="paragraph">
<p>Build <a href="#userland-content">userland programs</a>.</p>
</div>
<div class="paragraph">
<p>Build all with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland</pre>
</div>
</div>
<div class="paragraph">
<p>or build only those under e.g. <code>userland/c</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland userland/c</pre>
</div>
</div>
<div class="paragraph">
<p>The executables are not automatically added to the Buildroot image, you must follow the command with a <code>./build-buildroot</code> command as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland
./build-buildroot</pre>
</div>
</div>
<div class="paragraph">
<p>Remember that certain executables have specific requirements, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/">userland/arch/</a> programs only build if the target arch matches</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs">userland/libs</a> directory require the <code>--package</code> option <a href="#userland-libs-directory">userland/libs directory</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Default: build all examples that have their package dependencies met, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>an OpenBLAS example can only be built if the target root filesystem has the OpenBLAS libraries and headers installed, which you must inform with --package</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="c"><a class="anchor" href="#c"></a><a class="link" href="#c">27.2. C</a></h3>
<div class="paragraph">
<p>Programs under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/">userland/c/</a> are examples of <a href="https://en.wikipedia.org/wiki/ANSI_C">ANSI C</a> programming:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/empty.c">userland/c/empty.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a></p>
</li>
<li>
<p><code>main</code> and environment</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return0.c">userland/c/return0.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return1.c">userland/c/return1.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return2.c">userland/c/return2.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit0.c">userland/c/exit0.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit1.c">userland/c/exit1.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit2.c">userland/c/exit2.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/command_line_arguments.c">userland/c/command_line_arguments.c</a>: print one command line argument per line using <code>argc</code> and <code>argv</code>.</p>
<div class="paragraph">
<p>Good sanity check for user mode: <a href="#qemu-user-mode-getting-started">QEMU user mode getting started</a></p>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>Standard library</p>
<div class="ulist">
<ul>
<li>
<p><code>assert.h</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/assert_fail.c">userland/c/assert_fail.c</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>stdlib.h</code></p>
<div class="ulist">
<ul>
<li>
<p>exit</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/abort.c">userland/c/abort.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p><code>stdio.h</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/getchar.c">userland/c/getchar.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/snprintf.c">userland/c/snprintf.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/stderr.c">userland/c/stderr.c</a></p>
</li>
<li>
<p>File IO</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/file_write_read.c">userland/c/file_write_read.c</a></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/cat.c">userland/c/cat.c</a>: a quick and dirty <code>cat</code> implementation for interactive <a href="#user-mode-simulation">User mode simulation</a> tests</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/open_o_tmpfile.c">userland/linux/open_o_tmpfile.c</a>: <a href="https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030" class="bare">https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p><code>time.h</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/timespec_get.c">userland/c/timespec_get.c</a> <code>timespec_get</code> is a C11 for <code>clock_gettime</code> <a href="http://stackoverflow.com/questions/361363/how-to-measure-time-in-milliseconds-using-ansi-c/36095407#36095407" class="bare">http://stackoverflow.com/questions/361363/how-to-measure-time-in-milliseconds-using-ansi-c/36095407#36095407</a></p>
<div class="paragraph">
<p>Vs <code>clock()</code>: <a href="http://stackoverflow.com/questions/12392278/measure-time-in-linux-getrusage-vs-clock-gettime-vs-clock-vs-gettimeofday" class="bare">http://stackoverflow.com/questions/12392278/measure-time-in-linux-getrusage-vs-clock-gettime-vs-clock-vs-gettimeofday</a></p>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>Fun</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/loop.c">userland/c/loop.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="malloc"><a class="anchor" href="#malloc"></a><a class="link" href="#malloc">27.2.1. malloc</a></h4>
<div class="paragraph">
<p>Allocate memory! Vs using the stack: <a href="https://stackoverflow.com/questions/4584089/what-is-the-function-of-the-push-pop-instructions-used-on-registers-in-x86-ass/33583134#33583134" class="bare">https://stackoverflow.com/questions/4584089/what-is-the-function-of-the-push-pop-instructions-used-on-registers-in-x86-ass/33583134#33583134</a></p>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/malloc.c">userland/c/malloc.c</a>: <code>malloc</code> hello world: allocate two ints and use them.</p>
</div>
<div class="paragraph">
<p>Linux 5.1 / glibc 2.29 implements it with the <a href="#mmap"><code>mmap</code> system call</a>.</p>
</div>
<div class="paragraph">
<p><code>malloc</code> leads to the infinite joys of <a href="#memory-leaks">Memory leaks</a>.</p>
</div>
<div class="sect4">
<h5 id="malloc-implementation"><a class="anchor" href="#malloc-implementation"></a><a class="link" href="#malloc-implementation">27.2.1.1. malloc implementation</a></h5>
<div class="paragraph">
<p>TODO: the exact answer is going to be hard.</p>
</div>
<div class="paragraph">
<p>But at least let&#8217;s verify that large <code>malloc</code> calls use the <code>mmap</code> syscall with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>strace -x ./c/malloc_size.out 0x100000 2&gt;&amp;1 | grep mmap | tail -n 1
strace -x ./c/malloc_size.out 0x200000 2&gt;&amp;1 | grep mmap | tail -n 1
strace -x ./c/malloc_size.out 0x400000 2&gt;&amp;1 | grep mmap | tail -n 1</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/malloc_size.c">userland/c/malloc_size.c</a>.</p>
</div>
<div class="paragraph">
<p>From this we sese that the last <code>mmap</code> calls are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7ef2000
mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7271000
mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7071000</pre>
</div>
</div>
<div class="paragraph">
<p>which in hex are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf '%x\n' 1052672
# 101000
printf '%x\n' 2101248
# 201000
printf '%x\n' 4198400
# 401000</pre>
</div>
</div>
<div class="paragraph">
<p>so we figured out the pattern: those 1, 2, and 4 MiB mallocs are mmaping N + 0x1000 bytes.</p>
</div>
</div>
<div class="sect4">
<h5 id="malloc-maximum-size"><a class="anchor" href="#malloc-maximum-size"></a><a class="link" href="#malloc-maximum-size">27.2.1.2. malloc maximum size</a></h5>
<div class="paragraph">
<p>General overview at: <a href="https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate" class="bare">https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate</a></p>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/13127855/what-is-the-size-limit-for-mmap" class="bare">https://stackoverflow.com/questions/13127855/what-is-the-size-limit-for-mmap</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/7504139/malloc-allocates-memory-more-than-ram" class="bare">https://stackoverflow.com/questions/7504139/malloc-allocates-memory-more-than-ram</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>From <a href="#memory-size">Memory size</a> and <code>./run --help</code>, we see that at we set the emulator memory by default to 256MB. Let&#8217;s see how much Linux allows us to malloc.</p>
</div>
<div class="paragraph">
<p>Then from <a href="#malloc-implementation">malloc implementation</a> we see that <code>malloc</code> is implemented with <code>mmap</code>. Therefore, let&#8217;s simplify the problam and try to understand what is the larges mmap we can do first. This way we can ignore how glibc implements malloc for now.</p>
</div>
<div class="paragraph">
<p>In Linux, the maximum <code>mmap</code> value in controlled by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat /proc/sys/vm/overcommit_memory</pre>
</div>
</div>
<div class="paragraph">
<p>which is documented in <code>man proc</code>.</p>
</div>
<div class="paragraph">
<p>The default value is <code>0</code>, which I can&#8217;t find a precise documentation for. <code>2</code> is precisely documented but I&#8217;m lazy to do all calculations. So let&#8217;s just verify <code>0</code> vs <code>1</code> by trying to <code>mmap</code> 1GiB of memory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 0 &gt; /proc/sys/vm/overcommit_memory
./linux/mmap_anonymous.out 0x40000000
echo 1 &gt; /proc/sys/vm/overcommit_memory
./linux/mmap_anonymous.out 0x40000000</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/mmap_anonymous.c">userland/linux/mmap_anonymous.c</a></p>
</div>
<div class="paragraph">
<p>With <code>0</code>, we get a failure:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mmap: Cannot allocate memory</pre>
</div>
</div>
<div class="paragraph">
<p>but with <code>1</code> the allocation works.</p>
</div>
<div class="paragraph">
<p>We are allowed to allocate more than the actual memory + swap because the memory is only virtual, as explained at: <a href="https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management/57453334#57453334" class="bare">https://stackoverflow.com/questions/7880784/what-is-rss-and-vsz-in-linux-memory-management/57453334#57453334</a></p>
</div>
<div class="paragraph">
<p>If we start using the pages, the OOM killer would sooner or later step in and kill our process: <a href="#linux-out-of-memory-killer">Linux out-of-memory killer</a>.</p>
</div>
<div class="sect5">
<h6 id="linux-out-of-memory-killer"><a class="anchor" href="#linux-out-of-memory-killer"></a><a class="link" href="#linux-out-of-memory-killer">27.2.1.2.1. Linux out-of-memory killer</a></h6>
<div class="paragraph">
<p>We can observe the OOM in LKMC 1e969e832f66cb5a72d12d57c53fb09e9721d589 which defaults to 256MiB of memory with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo 1 &gt; /proc/sys/vm/overcommit_memory
./linux/mmap_anonymous_touch.out 0x40000000 0x8000000</pre>
</div>
</div>
<div class="paragraph">
<p>This first allows memory overcommit so to that the program can mmap 1GiB, 4x more than total RAM without failing as mentioned at <a href="#malloc-maximum-size">malloc maximum size</a>.</p>
</div>
<div class="paragraph">
<p>It then walks over every page and writes a value in it to ensure that it is used.</p>
</div>
<div class="paragraph">
<p>A <a href="#fork-bomb">Fork bomb</a> is another example that can trigger the OOM killer.</p>
</div>
<div class="paragraph">
<p>Algorithm used by the OOM: <a href="https://unix.stackexchange.com/questions/153585/how-does-the-oom-killer-decide-which-process-to-kill-first" class="bare">https://unix.stackexchange.com/questions/153585/how-does-the-oom-killer-decide-which-process-to-kill-first</a></p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="c-multithreading"><a class="anchor" href="#c-multithreading"></a><a class="link" href="#c-multithreading">27.2.2. C multithreading</a></h4>
<div class="paragraph">
<p>Added in C11!</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#userland-multithreading">Userland multithreading</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/3908031/how-to-multithread-c-code/52453354#52453354" class="bare">https://stackoverflow.com/questions/3908031/how-to-multithread-c-code/52453354#52453354</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="atomic-c"><a class="anchor" href="#atomic-c"></a><a class="link" href="#atomic-c">27.2.2.1. atomic.c</a></h5>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic.c">userland/c/atomic.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic/">userland/c/atomic/</a>: files in this directory use the same technique as <a href="#atomic-cpp">atomic.cpp</a>, i.e. with one special case per file.</p>
<div class="paragraph">
<p>Maybe <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic.c">userland/c/atomic.c</a> should be deprecated in favor of those more minimal ones.</p>
</div>
<div class="paragraph">
<p>This was added because C++-pre main is too bloated, especially when we turn one a gazillion <a href="#gem5">gem5</a> logs, it makes me want to cry.</p>
</div>
<div class="paragraph">
<p>And we want a single operation per test rather than to as in <code>atomic.c</code> because when using gem5 we want absolute control over the microbenchmark.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Demonstrates <code>atomic_int</code> and <code>thrd_create</code>.</p>
</div>
<div class="paragraph">
<p><a href="#disas">Disassembly with GDB</a> at LKMC 619fef4b04bddc4a5a38aec5e207dd4d5a25d206 + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --userland userland/c/atomic.c my_thread_main</pre>
</div>
</div>
<div class="paragraph">
<p>shows on ARM:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>16              ++cnt;
   0x00000000004008cc &lt;+28&gt;:    80 00 00 b0     adrp    x0, 0x411000 &lt;malloc@got.plt&gt;
   0x00000000004008d0 &lt;+32&gt;:    00 80 01 91     add     x0, x0, #0x60
   0x00000000004008d4 &lt;+36&gt;:    00 00 40 b9     ldr     w0, [x0]
   0x00000000004008d8 &lt;+40&gt;:    01 04 00 11     add     w1, w0, #0x1
   0x00000000004008dc &lt;+44&gt;:    80 00 00 b0     adrp    x0, 0x411000 &lt;malloc@got.plt&gt;
   0x00000000004008e0 &lt;+48&gt;:    00 80 01 91     add     x0, x0, #0x60
   0x00000000004008e4 &lt;+52&gt;:    01 00 00 b9     str     w1, [x0]

17              ++acnt;
   0x00000000004008e8 &lt;+56&gt;:    20 00 80 52     mov     w0, #0x1                        // #1
   0x00000000004008ec &lt;+60&gt;:    e0 1b 00 b9     str     w0, [sp, #24]
   0x00000000004008f0 &lt;+64&gt;:    e0 1b 40 b9     ldr     w0, [sp, #24]
   0x00000000004008f4 &lt;+68&gt;:    e2 03 00 2a     mov     w2, w0
   0x00000000004008f8 &lt;+72&gt;:    80 00 00 b0     adrp    x0, 0x411000 &lt;malloc@got.plt&gt;
   0x00000000004008fc &lt;+76&gt;:    00 70 01 91     add     x0, x0, #0x5c
   0x0000000000400900 &lt;+80&gt;:    03 00 e2 b8     ldaddal w2, w3, [x0]
   0x0000000000400904 &lt;+84&gt;:    61 00 02 0b     add     w1, w3, w2
   0x0000000000400908 &lt;+88&gt;:    e0 03 01 2a     mov     w0, w1
   0x000000000040090c &lt;+92&gt;:    e0 1f 00 b9     str     w0, [sp, #28]</pre>
</div>
</div>
<div class="paragraph">
<p>so:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the atomic increment uses <a href="#arm-lse"><code>ldadd</code></a></p>
</li>
<li>
<p>the non-atomic increment just does LDR, ADD, STR: <a href="#arm-lse"><code>ldadd</code></a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>With <code>-O3</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>16              ++cnt;
   0x0000000000400a00 &lt;+32&gt;:    60 00 40 b9     ldr     w0, [x3]
   0x0000000000400a04 &lt;+36&gt;:    00 04 00 11     add     w0, w0, #0x1
   0x0000000000400a08 &lt;+40&gt;:    60 00 00 b9     str     w0, [x3]

17              ++acnt;
   0x0000000000400a0c &lt;+44&gt;:    20 00 80 52     mov     w0, #0x1                        // #1
   0x0000000000400a10 &lt;+48&gt;:    40 00 e0 b8     ldaddal w0, w0, [x2]</pre>
</div>
</div>
<div class="paragraph">
<p>so the situation is the same but without all the horrible stack noise.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gcc-c-extensions"><a class="anchor" href="#gcc-c-extensions"></a><a class="link" href="#gcc-c-extensions">27.2.3. GCC C extensions</a></h4>
<div class="sect4">
<h5 id="c-empty-struct"><a class="anchor" href="#c-empty-struct"></a><a class="link" href="#c-empty-struct">27.2.3.1. C empty struct</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/empty_struct.c">userland/gcc/empty_struct.c</a></p>
</div>
<div class="paragraph">
<p>Documentation: <a href="https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Empty-Structures.html#Empty-Structures" class="bare">https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Empty-Structures.html#Empty-Structures</a></p>
</div>
<div class="paragraph">
<p>Question: <a href="https://stackoverflow.com/questions/24685399/c-empty-struct-what-does-this-mean-do" class="bare">https://stackoverflow.com/questions/24685399/c-empty-struct-what-does-this-mean-do</a></p>
</div>
</div>
<div class="sect4">
<h5 id="openmp"><a class="anchor" href="#openmp"></a><a class="link" href="#openmp">27.2.3.2. OpenMP</a></h5>
<div class="paragraph">
<p>GCC implements the <a href="#openmp">OpenMP</a> threading implementation: <a href="https://stackoverflow.com/questions/3949901/pthreads-vs-openmp" class="bare">https://stackoverflow.com/questions/3949901/pthreads-vs-openmp</a></p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/openmp.c">userland/gcc/openmp.c</a></p>
</div>
<div class="paragraph">
<p>The implementation is built into GCC itself. It is enabled at GCC compile time by <code>BR2_GCC_ENABLE_OPENMP=y</code> on Buildroot, and at program compile time by <code>-fopenmp</code>.</p>
</div>
<div class="paragraph">
<p>It seems to be easier to use for compute parallelism and more language agnostic than POSIX threads.</p>
</div>
<div class="paragraph">
<p>pthreads are more versatile though and allow for a superset of OpenMP.</p>
</div>
<div class="paragraph">
<p>The implementation lives under <code>libgomp</code> in the GCC tree, and is documented at: <a href="https://gcc.gnu.org/onlinedocs/libgomp/" class="bare">https://gcc.gnu.org/onlinedocs/libgomp/</a></p>
</div>
<div class="paragraph">
<p><code>strace</code> shows that OpenMP makes <code>clone()</code> syscalls in Linux. TODO: does it actually call <code>pthread_</code> functions, or does it make syscalls directly? Or in other words, can it work on <a href="#freestanding-programs">Freestanding programs</a>? A quick grep shows many references to pthreads.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="cpp"><a class="anchor" href="#cpp"></a><a class="link" href="#cpp">27.3. C++</a></h3>
<div class="paragraph">
<p>Programs under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/">userland/cpp/</a> are examples of <a href="https://en.wikipedia.org/wiki/C%2B%2B#Standardization">ISO C</a> programming.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/empty.cpp">userland/cpp/empty.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/hello.cpp">userland/cpp/hello.cpp</a></p>
</li>
<li>
<p>iostream</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/copyfmt.cpp">userland/cpp/copyfmt.cpp</a>: <code>std::copyfmt</code> restores stream state, see also: <a href="https://stackoverflow.com/questions/12560291/set-back-default-floating-point-print-precision-in-c/53673686#53673686" class="bare">https://stackoverflow.com/questions/12560291/set-back-default-floating-point-print-precision-in-c/53673686#53673686</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>fstream</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/file_write_read.cpp">userland/cpp/file_write_read.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/temporary_directory.cpp">userland/cpp/temporary_directory.cpp</a>: illustrates <code>std::filesystem::temp_directory_path</code> and answers <a href="https://stackoverflow.com/questions/3379956/how-to-create-a-temporary-directory-in-c/58454949#58454949" class="bare">https://stackoverflow.com/questions/3379956/how-to-create-a-temporary-directory-in-c/58454949#58454949</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>random</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/random.cpp">userland/cpp/random.cpp</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>containers</p>
<div class="ulist">
<ul>
<li>
<p>associative</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/set.cpp">userland/cpp/set.cpp</a>: <code>std::set</code> contains unique keys</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/map.cpp">userland/cpp/map.cpp</a>: <code>std::map</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/multimap.cpp">userland/cpp/multimap.cpp</a>: <code>std::multimap</code></p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="#algorithms">Algorithms</a> contains a benchmark comparison of different c++ containers</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="cpp-classes"><a class="anchor" href="#cpp-classes"></a><a class="link" href="#cpp-classes">27.3.1. C++ classes</a></h4>
<div class="sect4">
<h5 id="cpp-constructor"><a class="anchor" href="#cpp-constructor"></a><a class="link" href="#cpp-constructor">27.3.1.1. C++ constructor</a></h5>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/initializer_list_constructor.cpp">userland/cpp/initializer_list_constructor.cpp</a>: documents stuff like <code>std::vector&lt;int&gt; v{0, 1};</code> and <code>std::initializer_list</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/most_vexing_parse.cpp">userland/cpp/most_vexing_parse.cpp</a>: the most vexing parse is a famous constructor vs function declaration syntax gotcha!</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/Most_vexing_parse" class="bare">https://en.wikipedia.org/wiki/Most_vexing_parse</a></p>
</li>
<li>
<p><a href="http://stackoverflow.com/questions/180172/default-constructor-with-empty-brackets" class="bare">http://stackoverflow.com/questions/180172/default-constructor-with-empty-brackets</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>virtual</code> and polymorphism</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/virtual.cpp">userland/cpp/virtual.cpp</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="cpp-rule-of-five"><a class="anchor" href="#cpp-rule-of-five"></a><a class="link" href="#cpp-rule-of-five">27.3.1.1.1. C++ rule of five</a></h6>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/rule_of_five.cpp">userland/cpp/rule_of_five.cpp</a></p>
</div>
<div class="paragraph">
<p>Output Ubuntu 20.04 GCC 9.3:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>constructor?
constructor

copy?
constructor
copy

copy assignment?
constructor
copy assignment
constructor
copy
move assignment
destructor

move?
constructor

move?
constructor
constructor
move assignment
destructor

a bunch of destructors?
destructor
destructor
destructor
destructor
destructor</pre>
</div>
</div>
<div class="paragraph">
<p><a href="https://en.cppreference.com/w/cpp/language/rule_of_three" class="bare">https://en.cppreference.com/w/cpp/language/rule_of_three</a></p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="cpp-standards"><a class="anchor" href="#cpp-standards"></a><a class="link" href="#cpp-standards">27.3.2. C++ standards</a></h4>
<div class="paragraph">
<p>Like for C, you have to pay for the standards&#8230;&#8203; insane. So we just use the closest free drafts instead.</p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents" class="bare">https://stackoverflow.com/questions/81656/where-do-i-find-the-current-c-or-c-standard-documents</a></p>
</div>
</div>
<div class="sect3">
<h4 id="cpp-initialization-types"><a class="anchor" href="#cpp-initialization-types"></a><a class="link" href="#cpp-initialization-types">27.3.3. C++ initialization types</a></h4>
<div class="paragraph">
<p>OMG this is hell, understand when primitive variables are initialized or not:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/3127454/how-do-c-class-members-get-initialized-if-i-dont-do-it-explicitly" class="bare">https://stackoverflow.com/questions/3127454/how-do-c-class-members-get-initialized-if-i-dont-do-it-explicitly</a></p>
</li>
<li>
<p><a href="https://blog.tartanllama.xyz/initialization-is-bonkers/" class="bare">https://blog.tartanllama.xyz/initialization-is-bonkers/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Intuition:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>direct initialization: a constructor called explicitly with at least one argument: <a href="https://en.cppreference.com/w/cpp/language/direct_initialization" class="bare">https://en.cppreference.com/w/cpp/language/direct_initialization</a></p>
</li>
<li>
<p>default initialization: does not initialize primitive types: <a href="https://en.cppreference.com/w/cpp/language/default_initialization" class="bare">https://en.cppreference.com/w/cpp/language/default_initialization</a></p>
</li>
<li>
<p>value initialization: maybe initializes primitive types: <a href="https://en.cppreference.com/w/cpp/language/value_initialization" class="bare">https://en.cppreference.com/w/cpp/language/value_initialization</a></p>
</li>
<li>
<p>zero initialization: initializes primitive types</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Good rule:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>initialize every single variable explicitly to prevent the risk of having uninitialized variables due to programmer error (which is easy to get wrong due to insane rules)</p>
</li>
<li>
<p>if you don&#8217;t define your own default constructor, always <code>= delete</code> it instead. This prevents the possibility that variables will be assigned twice due to zero initialization</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="cpp-multithreading"><a class="anchor" href="#cpp-multithreading"></a><a class="link" href="#cpp-multithreading">27.3.4. C++ multithreading</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.cppreference.com/w/cpp/header/thread"><code>&lt;thread&gt;</code></a></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/count.cpp">userland/cpp/count.cpp</a> Exemplifies: <code>std::this_thread::sleep_for</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/thread_hardware_concurrency.cpp">userland/cpp/thread_hardware_concurrency.cpp</a> <code>std::thread::hardware_concurrency</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/thread_get_id.cpp">userland/cpp/thread_get_id.cpp</a> <code>std::thread::get_id</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/thread_return_value.cpp">userland/cpp/thread_return_value.cpp</a>: how to return a value from a thread</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://en.cppreference.com/w/cpp/header/atomic"><code>&lt;atomic&gt;</code></a>: <a href="#cpp17">C++17 N4659 standards draft</a> 32 "Atomic operations library"</p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="atomic-cpp"><a class="anchor" href="#atomic-cpp"></a><a class="link" href="#atomic-cpp">27.3.4.1. atomic.cpp</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/">userland/cpp/atomic/</a></p>
</div>
<div class="paragraph">
<p>C version at: <a href="#atomic-c">atomic.c</a>.</p>
</div>
<div class="paragraph">
<p>In this set of examples, we exemplify various synchronization mechanisms, including assembly specific ones, by using the convenience of C++ multithreading:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/main.hpp">userland/cpp/atomic/main.hpp</a>: contains all the code which is then specialized in separated <code>.cpp</code> files with macros</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_add.cpp">userland/cpp/atomic/aarch64_add.cpp</a>: non synchronized aarch64 inline assembly</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldaxr_stlxr.cpp">userland/cpp/atomic/aarch64_ldaxr_stlxr.cpp</a>: see: <a href="#arm-ldxr-and-stxr-instructions">ARM LDXR and STXR instructions</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldadd.cpp">userland/cpp/atomic/aarch64_ldadd.cpp</a>: synchronized aarch64 inline assembly with the <a href="#arm-lse">ARM Large System Extensions (LSE)</a> LDADD instruction</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/fail.cpp">userland/cpp/atomic/fail.cpp</a>: non synchronized C operator ``</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/mutex.cpp">userland/cpp/atomic/mutex.cpp</a>: synchronized <code>std::mutex</code>. <code>std;</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/std_atomic.cpp">userland/cpp/atomic/std_atomic.cpp</a>: synchronized <code>std::atomic_ulong</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_inc.cpp">userland/cpp/atomic/x86_64_inc.cpp</a>: non synchronized x86_64 inline assembly</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_lock_inc.cpp">userland/cpp/atomic/x86_64_lock_inc.cpp</a>: synchronized x86_64 inline assembly with the <a href="#x86-lock-prefix">x86 LOCK prefix</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>All examples do exactly the same thing: span N threads and loop M times in each thread incrementing a global integer.</p>
</div>
<div class="paragraph">
<p>For inputs large enough, the non-synchronized examples are extremely likely to produce "wrong" results, for example on <a href="#p51">2017 Lenovo ThinkPad P51</a> Ubuntu 19.10 <a href="#userland-setup-getting-started-natively">native</a> with 2 threads and 10000 loops:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./fail.out 2 10000</pre>
</div>
</div>
<div class="paragraph">
<p>we could get an output such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>expect 20000
global 12676</pre>
</div>
</div>
<div class="paragraph">
<p>The actual value is much smaller, because the threads have often overwritten one another with older values.</p>
</div>
<div class="paragraph">
<p>With <a href="#optimization-level-of-a-build"><code>--optimization-level 3</code></a>, the result almost always equals that of a single thread, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --optimization-level 3 --force-rebuild fail.cpp
./fail.out 4 1000000</pre>
</div>
</div>
<div class="paragraph">
<p>usually gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>expect 40000
global 10000</pre>
</div>
</div>
<div class="paragraph">
<p>This is because now, instead of the horribly inefficient <code>-O0</code> assembly that reads <code>global</code> from memory every time, the code:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>reads <code>global</code> to a register</p>
</li>
<li>
<p>increments the register</p>
</li>
<li>
<p>at end the end, the resulting value of each thread gets written back, overwriting each other with the increment of each thread</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The <code>-O0</code> code therefore mixes things up much more because it reads and write back to memory many many times.</p>
</div>
<div class="paragraph">
<p>This can be easily seen from the disassembly with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gdb -batch -ex "disassemble threadMain" fail.out</pre>
</div>
</div>
<div class="paragraph">
<p>which gives for <code>-O0</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x0000000000402656 &lt;+0&gt;:     endbr64
   0x000000000040265a &lt;+4&gt;:     push   %rbp
   0x000000000040265b &lt;+5&gt;:     mov    %rsp,%rbp
   0x000000000040265e &lt;+8&gt;:     movq   $0x0,-0x8(%rbp)
   0x0000000000402666 &lt;+16&gt;:    mov    0x5c2b(%rip),%rax        # 0x408298 &lt;niters&gt;
   0x000000000040266d &lt;+23&gt;:    cmp    %rax,-0x8(%rbp)
   0x0000000000402671 &lt;+27&gt;:    jae    0x40269b &lt;threadMain()+69&gt;
   0x0000000000402673 &lt;+29&gt;:    mov    0x5c26(%rip),%rdx        # 0x4082a0 &lt;global&gt;
   0x000000000040267a &lt;+36&gt;:    mov    -0x8(%rbp),%rax
   0x000000000040267e &lt;+40&gt;:    mov    %rax,-0x8(%rbp)
   0x0000000000402682 &lt;+44&gt;:    mov    0x5c17(%rip),%rax        # 0x4082a0 &lt;global&gt;
   0x0000000000402689 &lt;+51&gt;:    add    $0x1,%rax
   0x000000000040268d &lt;+55&gt;:    mov    %rax,0x5c0c(%rip)        # 0x4082a0 &lt;global&gt;
   0x0000000000402694 &lt;+62&gt;:    addq   $0x1,-0x8(%rbp)
   0x0000000000402699 &lt;+67&gt;:    jmp    0x402666 &lt;threadMain()+16&gt;
   0x000000000040269b &lt;+69&gt;:    nop
   0x000000000040269c &lt;+70&gt;:    pop    %rbp
   0x000000000040269d &lt;+71&gt;:    retq</pre>
</div>
</div>
<div class="paragraph">
<p>and for <code>-O3</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x00000000004017f0 &lt;+0&gt;:     endbr64
   0x00000000004017f4 &lt;+4&gt;:     mov    0x2a25(%rip),%rcx        # 0x404220 &lt;niters&gt;
   0x00000000004017fb &lt;+11&gt;:    test   %rcx,%rcx
   0x00000000004017fe &lt;+14&gt;:    je     0x401824 &lt;threadMain()+52&gt;
   0x0000000000401800 &lt;+16&gt;:    mov    0x2a11(%rip),%rdx        # 0x404218 &lt;global&gt;
   0x0000000000401807 &lt;+23&gt;:    xor    %eax,%eax
   0x0000000000401809 &lt;+25&gt;:    nopl   0x0(%rax)
   0x0000000000401810 &lt;+32&gt;:    add    $0x1,%rax
   0x0000000000401814 &lt;+36&gt;:    add    $0x1,%rdx
   0x0000000000401818 &lt;+40&gt;:    cmp    %rcx,%rax
   0x000000000040181b &lt;+43&gt;:    jb     0x401810 &lt;threadMain()+32&gt;
   0x000000000040181d &lt;+45&gt;:    mov    %rdx,0x29f4(%rip)        # 0x404218 &lt;global&gt;
   0x0000000000401824 &lt;+52&gt;:    retq</pre>
</div>
</div>
<div class="paragraph">
<p>We can now look into how <code>std::atomic</code> is implemented. In <code>-O3</code> the disassembly is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x0000000000401770 &lt;+0&gt;:     endbr64
   0x0000000000401774 &lt;+4&gt;:     cmpq   $0x0,0x297c(%rip)        # 0x4040f8 &lt;niters&gt;
   0x000000000040177c &lt;+12&gt;:    je     0x401796 &lt;threadMain()+38&gt;
   0x000000000040177e &lt;+14&gt;:    xor    %eax,%eax
   0x0000000000401780 &lt;+16&gt;:    lock addq $0x1,0x2967(%rip)        # 0x4040f0 &lt;global&gt;
   0x0000000000401789 &lt;+25&gt;:    add    $0x1,%rax
   0x000000000040178d &lt;+29&gt;:    cmp    %rax,0x2964(%rip)        # 0x4040f8 &lt;niters&gt;
   0x0000000000401794 &lt;+36&gt;:    ja     0x401780 &lt;threadMain()+16&gt;
   0x0000000000401796 &lt;+38&gt;:    retq</pre>
</div>
</div>
<div class="paragraph">
<p>so we clearly see that basically a <code>lock addq</code> is used to do an atomic read and write to memory every single time, just like in our other example <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_lock_inc.cpp">userland/cpp/atomic/x86_64_lock_inc.cpp</a>.</p>
</div>
<div class="paragraph">
<p>This setup can also be used to benchmark different synchronization mechanisms. For example, <code>std::mutex</code> was about 1.5x slower with two cores than <code>std::atomic</code>, presumably because it relies on the <a href="#futex-system-call"><code>futex</code> system call</a> as can be seen from <code>strace -f -s999 -v</code> logs, while <code>std::atomic</code> uses just userland instructions: <a href="https://www.quora.com/How-does-std-atomic-work-in-C++11/answer/Ciro-Santilli" class="bare">https://www.quora.com/How-does-std-atomic-work-in-C++11/answer/Ciro-Santilli</a> Tested in <code>-O3</code> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>time ./std_atomic.out 4 100000000
time ./mutex.out 4 100000000</pre>
</div>
</div>
<div class="paragraph">
<p>Related examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>POSIX <a href="#pthread-mutex">pthread_mutex</a></p>
</li>
<li>
<p>C11 <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic.c">userland/c/atomic.c</a> documented at <a href="#c-multithreading">C multithreading</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/31978324/what-exactly-is-stdatomic/58904448#58904448" class="bare">https://stackoverflow.com/questions/31978324/what-exactly-is-stdatomic/58904448#58904448</a> "What exactly is std::atomic?"</p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="detailed-gem5-analysis-of-how-data-races-happen"><a class="anchor" href="#detailed-gem5-analysis-of-how-data-races-happen"></a><a class="link" href="#detailed-gem5-analysis-of-how-data-races-happen">27.3.4.1.1. Detailed gem5 analysis of how data races happen</a></h6>
<div class="paragraph">
<p>The smallest data race we managed to come up as of LKMC 7c01b29f1ee7da878c7cc9cb4565f3f3cf516a92 and gem5 872cb227fdc0b4d60acc7840889d567a6936b6e1 was with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/atomic.c">userland/c/atomic.c</a> (see also <a href="#c-multithreading">C multithreading</a>):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --cli-args '2 10' \
  --cpus 3 \
  --emulator gem5 \
  --userland userland/c/atomic.c \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>atomic 20
non-atomic 19</pre>
</div>
</div>
<div class="paragraph">
<p>Note that that the system is very minimal, and doesn&#8217;t even have caches, so I&#8217;m curious as to how this can happen at all.</p>
</div>
<div class="paragraph">
<p>So first we do a run with <a href="#gem5-tracing"><code>--trace Exec</code></a> and look at the <code>my_thread_main</code> entries.</p>
</div>
<div class="paragraph">
<p>From there we see that first CPU1 enters the function, since it was spawned first.</p>
</div>
<div class="paragraph">
<p>Then for some time, both CPU1 and CPU2 are running at the same time.</p>
</div>
<div class="paragraph">
<p>Finally, CPU1 exists, then CPU2 runs alone for a while to finish its loops, and then CPU2 exits.</p>
</div>
<div class="paragraph">
<p>By greping the LDR data read from the log, we are able to easily spot the moment where things started to go wrong based on the <code>D=</code> data:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>grep -E 'my_thread_main\+36' trace.txt &gt; trace-ldr.txt</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>grep</code> output contains</p>
</div>
<div class="literalblock">
<div class="content">
<pre>94024500: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000006 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94036500: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000007 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94048500: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000008 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94058500: system.cpu2: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94060500: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94070500: system.cpu2: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x000000000000000a A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94082500: system.cpu2: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x000000000000000b A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)</pre>
</div>
</div>
<div class="paragraph">
<p>and so se see that it is at <code>94058500</code> that things started going bad, since two consecutive loads from different CPUs read the same value <code>D=9</code>! Actually, things were not too bad afterwards because this was by coincidence the last CPU1 read, we would have missed many more increments if the number of iterations had been larger.</p>
</div>
<div class="paragraph">
<p>Now that we have the first bad time, let&#8217;s look at the fuller disassembly to better understand what happens around that point.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>94058500: system.cpu2: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94059000: system.cpu2: A0 T0 : @my_thread_main+40    :   add   w1, w0, #1         : IntAlu :  D=0x000000000000000a  flags=(IsInteger)
94059000: system.cpu1: A0 T0 : @my_thread_main+120    :   b.cc   &lt;my_thread_main+28&gt; : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
94059500: system.cpu1: A0 T0 : @my_thread_main+28    :   adrp   x0, #69632        : IntAlu :  D=0x0000000000411000  flags=(IsInteger)
94059500: system.cpu2: A0 T0 : @my_thread_main+44    :   adrp   x0, #69632        : IntAlu :  D=0x0000000000411000  flags=(IsInteger)
94060000: system.cpu2: A0 T0 : @my_thread_main+48    :   add   x0, x0, #96        : IntAlu :  D=0x0000000000411060  flags=(IsInteger)
94060000: system.cpu1: A0 T0 : @my_thread_main+32    :   add   x0, x0, #96        : IntAlu :  D=0x0000000000411060  flags=(IsInteger)
94060500: system.cpu1: A0 T0 : @my_thread_main+36    :   ldr   x0, [x0]           : MemRead :  D=0x0000000000000009 A=0x411060  flags=(IsInteger|IsMemRef|IsLoad)
94060500: system.cpu2: A0 T0 : @my_thread_main+52    :   str   x1, [x0]           : MemWrite :  D=0x000000000000000a A=0x411060  flags=(IsInteger|IsMemRef|IsStore)</pre>
</div>
</div>
<div class="paragraph">
<p>and from this, all becomes crystal clear:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>94058500: CPU2 loads</p>
</li>
<li>
<p>94060500: CPU1 loads</p>
</li>
<li>
<p>94060500: CPU2 stores</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so we see that CPU2 just happened to store after CPU1 loads.</p>
</div>
<div class="paragraph">
<p>We also understand why LDADD solves the race problem in AtomicSimpleCPU: it does the load and store in one single go!</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="cpp-memory-order"><a class="anchor" href="#cpp-memory-order"></a><a class="link" href="#cpp-memory-order">27.3.4.2. C++ std::memory_order</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean" class="bare">https://stackoverflow.com/questions/12346487/what-do-each-memory-order-mean</a></p>
</div>
<div class="paragraph">
<p>TODO let&#8217;s understand that fully one day.</p>
</div>
<div class="paragraph">
<p>This is the C++ version of the more general <a href="#memory-consistency">Memory consistency</a> concept.</p>
</div>
</div>
<div class="sect4">
<h5 id="cpp-parallel-algorithms"><a class="anchor" href="#cpp-parallel-algorithms"></a><a class="link" href="#cpp-parallel-algorithms">27.3.4.3. C++ parallel algorithms</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/51031060/are-c17-parallel-algorithms-implemented-already/55989883#55989883" class="bare">https://stackoverflow.com/questions/51031060/are-c17-parallel-algorithms-implemented-already/55989883#55989883</a></p>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/parallel_sort.cpp">userland/cpp/parallel_sort.cpp</a></p>
</div>
</div>
<div class="sect4">
<h5 id="cpp17"><a class="anchor" href="#cpp17"></a><a class="link" href="#cpp17">27.3.4.4. C++17 N4659 standards draft</a></h5>
<div class="paragraph">
<p><a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4659.pdf" class="bare">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/n4659.pdf</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="cpp-templates"><a class="anchor" href="#cpp-templates"></a><a class="link" href="#cpp-templates">27.3.5. C++ templates</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/template.cpp">userland/cpp/template.cpp</a>: basic example</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/template_class_with_static_member.cpp">userland/cpp/template_class_with_static_member.cpp</a>: <a href="https://stackoverflow.com/questions/3229883/static-member-initialization-in-a-class-template" class="bare">https://stackoverflow.com/questions/3229883/static-member-initialization-in-a-class-template</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="sfinae"><a class="anchor" href="#sfinae"></a><a class="link" href="#sfinae">27.3.5.1. SFINAE</a></h5>
<div class="paragraph">
<p><a href="https://en.cppreference.com/w/cpp/language/sfinae" class="bare">https://en.cppreference.com/w/cpp/language/sfinae</a></p>
</div>
<div class="paragraph">
<p>Not possible to do the typecheck automatically without explicitly giving type constraints: <a href="https://stackoverflow.com/questions/53441832/sfinae-automatically-check-that-function-body-compiles-without-explicit-constrai" class="bare">https://stackoverflow.com/questions/53441832/sfinae-automatically-check-that-function-body-compiles-without-explicit-constrai</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="cpp-type-casting"><a class="anchor" href="#cpp-type-casting"></a><a class="link" href="#cpp-type-casting">27.3.6. C++ type casting</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/static_dynamic_reinterpret_cast.cpp">userland/cpp/static_dynamic_reinterpret_cast.cpp</a></p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/332030/when-should-static-cast-dynamic-cast-const-cast-and-reinterpret-cast-be-used/60414256#60414256" class="bare">https://stackoverflow.com/questions/332030/when-should-static-cast-dynamic-cast-const-cast-and-reinterpret-cast-be-used/60414256#60414256</a></p>
</div>
</div>
<div class="sect3">
<h4 id="cpp-compile-time-magic"><a class="anchor" href="#cpp-compile-time-magic"></a><a class="link" href="#cpp-compile-time-magic">27.3.7. C++ compile time magic</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/if_constexpr.cpp">userland/cpp/if_constexpr.cpp</a>: C++17 <code>if constexpr</code>: <a href="https://stackoverflow.com/questions/12160765/if-else-at-compile-time-in-c/54647315#54647315" class="bare">https://stackoverflow.com/questions/12160765/if-else-at-compile-time-in-c/54647315#54647315</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="cpp-decltype"><a class="anchor" href="#cpp-decltype"></a><a class="link" href="#cpp-decltype">27.3.7.1. C++ <code>decltype</code></a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/decltype.cpp">userland/cpp/decltype.cpp</a></p>
</div>
<div class="paragraph">
<p>C++11 keyword.</p>
</div>
<div class="paragraph">
<p>Replaces decltype with type of an expression at compile time.</p>
</div>
<div class="paragraph">
<p>More powerful than <code>auto</code> as you can use it in more places.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="cpp-concepts"><a class="anchor" href="#cpp-concepts"></a><a class="link" href="#cpp-concepts">27.3.8. C++ concepts</a></h4>
<div class="sect4">
<h5 id="cpp-iterators"><a class="anchor" href="#cpp-iterators"></a><a class="link" href="#cpp-iterators">27.3.8.1. C++ iterators</a></h5>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/37031805/preparation-for-stditerator-being-deprecated/38103394" class="bare">https://stackoverflow.com/questions/37031805/preparation-for-stditerator-being-deprecated/38103394</a></p>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/custom_iterator.cpp">userland/cpp/custom_iterator.cpp</a>: there is no way to easily define a nice custom iterator, you just have to wrap existing iterators and add a gazillion wrapper methods:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/8054273/how-to-implement-an-stl-style-iterator-and-avoid-common-pitfalls" class="bare">https://stackoverflow.com/questions/8054273/how-to-implement-an-stl-style-iterator-and-avoid-common-pitfalls</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/3582608/how-to-correctly-implement-custom-iterators-and-const-iterators" class="bare">https://stackoverflow.com/questions/3582608/how-to-correctly-implement-custom-iterators-and-const-iterators</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/6471019/can-should-i-inherit-from-an-stl-iterator" class="bare">https://stackoverflow.com/questions/6471019/can-should-i-inherit-from-an-stl-iterator</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="cpp-third-party-libraries"><a class="anchor" href="#cpp-third-party-libraries"></a><a class="link" href="#cpp-third-party-libraries">27.3.9. C++ third-party libraries</a></h4>
<div class="paragraph">
<p>Under: <a href="#userland-libs-directory">userland/libs directory</a>.</p>
</div>
<div class="sect4">
<h5 id="boost"><a class="anchor" href="#boost"></a><a class="link" href="#boost">27.3.9.1. Boost</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Boost_(C%2B%2B_libraries)">https://en.wikipedia.org/wiki/Boost_(C%2B%2B_libraries)</a></p>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/boost">userland/libs/boost</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/boost/bimap.cpp">userland/libs/boost/bimap.cpp</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="googletest"><a class="anchor" href="#googletest"></a><a class="link" href="#googletest">27.3.9.2. GoogleTest</a></h5>
<div class="paragraph">
<p><a href="https://github.com/google/googletest" class="bare">https://github.com/google/googletest</a></p>
</div>
<div class="paragraph">
<p>On Ubuntu 20.04, the package:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt install googletest</pre>
</div>
</div>
<div class="paragraph">
<p>does not contain prebuilts, and it is intentional, it is incomprehensible:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://askubuntu.com/questions/97626/how-to-install-googletest/1295185#1295185" class="bare">https://askubuntu.com/questions/97626/how-to-install-googletest/1295185#1295185</a></p>
</li>
<li>
<p><a href="https://askubuntu.com/questions/145887/why-no-library-files-installed-for-google-test" class="bare">https://askubuntu.com/questions/145887/why-no-library-files-installed-for-google-test</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so you might as well just <code>git clone</code> and build the damned thing yourself:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git submodule update --init submodules/googletest
cd submodules/googletest
mkdir build
cd build
cmake ..
make -j`nproc`
cd ../../userland/libs/googletest
./build</pre>
</div>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/googletest">userland/libs/googletest</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>userland/libs/googletest/main.cpp[]</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="hdf5"><a class="anchor" href="#hdf5"></a><a class="link" href="#hdf5">27.3.9.3. HDF5</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Hierarchical_Data_Format" class="bare">https://en.wikipedia.org/wiki/Hierarchical_Data_Format</a></p>
</div>
<div class="paragraph">
<p>Binary format to store data. TODO vs databases, notably SQLite: <a href="https://datascience.stackexchange.com/questions/262/hierarchical-data-format-what-are-the-advantages-compared-to-alternative-format" class="bare">https://datascience.stackexchange.com/questions/262/hierarchical-data-format-what-are-the-advantages-compared-to-alternative-format</a></p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/hdf5">userland/libs/hdf5</a></p>
</li>
<li>
<p>gem5 can dump statistics as HDF5: <a href="#gem5-hdf5-statistics">gem5 HDF5 statistics</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="posix"><a class="anchor" href="#posix"></a><a class="link" href="#posix">27.4. POSIX</a></h3>
<div class="paragraph">
<p>Programs under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/">userland/posix/</a> are examples of POSIX C programming.</p>
</div>
<div class="paragraph">
<p>These links provide a clear overview of what POSIX is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/1780599/what-is-the-meaning-of-posix/31865755#31865755" class="bare">https://stackoverflow.com/questions/1780599/what-is-the-meaning-of-posix/31865755#31865755</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/11983/what-exactly-is-posix/220877#220877" class="bare">https://unix.stackexchange.com/questions/11983/what-exactly-is-posix/220877#220877</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="environment-variables"><a class="anchor" href="#environment-variables"></a><a class="link" href="#environment-variables">27.4.1. Environment variables</a></h4>
<div class="paragraph">
<p>POSIX C example that prints all environment variables: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/environ.c">userland/posix/environ.c</a></p>
</div>
</div>
<div class="sect3">
<h4 id="unistd-h"><a class="anchor" href="#unistd-h"></a><a class="link" href="#unistd-h">27.4.2. unistd.h</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count.c">userland/posix/count.c</a> illustrates <code>sleep()</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count_to.c">userland/posix/count_to.c</a> minor variation of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count.c">userland/posix/count.c</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="fork"><a class="anchor" href="#fork"></a><a class="link" href="#fork">27.4.3. fork</a></h4>
<div class="paragraph">
<p>POSIX' multiprocess API. Contrast with <a href="#pthreads">pthreads</a> which are for threads.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/fork.c">userland/posix/fork.c</a></p>
</div>
<div class="paragraph">
<p>Sample <a href="#userland-setup-getting-started-natively">native userland output</a> on Ubuntu 19.04 at 762cd8d601b7db06aa289c0fca7b40696299a868 + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>before fork before fork pid=13038 ppid=4805
after fork after fork pid=13038 ppid=4805
after (pid == 0) after (pid == 0) pid=13038 ppid=4805
after fork after fork pid=13039 ppid=13038
inside (pid == 0) inside (pid == 0) pid=13039 ppid=13038
after wait after wait pid=13038 ppid=4805
fork() return = 13039</pre>
</div>
</div>
<div class="paragraph">
<p>Read the source comments and understand everything that is going on!</p>
</div>
<div class="sect4">
<h5 id="getpid"><a class="anchor" href="#getpid"></a><a class="link" href="#getpid">27.4.3.1. getpid</a></h5>
<div class="paragraph">
<p>The minimal interesting example is to use fork and observe different PIDs.</p>
</div>
<div class="paragraph">
<p>A more minimal test-like example without forking can be seen at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/getpid.c">userland/posix/getpid.c</a>.</p>
</div>
<div class="paragraph">
<p>This example can for example be used used to play with: <a href="#gem5-syscall-emulation-multiple-executables">gem5 syscall emulation multiple executables</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="fork-bomb"><a class="anchor" href="#fork-bomb"></a><a class="link" href="#fork-bomb">27.4.3.2. Fork bomb</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Fork_bomb" class="bare">https://en.wikipedia.org/wiki/Fork_bomb</a></p>
</div>
<div class="paragraph">
<p>DANGER! Only run this on your host if you have saved all data you care about! Better run it inside an emulator! QEMU v4.0.0 <a href="#user-mode-simulation">user mode</a> is not safe enough either because it is very native does not limit guest memory, so it will still blow up the host!</p>
</div>
<div class="paragraph">
<p>So without further ado, let&#8217;s rock with either:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './posix/fork_bomb.out danger'
./run --eval-after './fork_bomb.sh danger'</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/fork_bomb.c">userland/posix/fork_bomb.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/fork_bomb.sh">rootfs_overlay/lkmc/fork_bomb.sh</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Outcome for the C version on LKMC 762cd8d601b7db06aa289c0fca7b40696299a868 + 1: after a few seconds of an unresponsive shell, we get a visit form the <a href="#linux-out-of-memory-killer">Linux out-of-memory killer</a>, and the system is restored!</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="pthreads"><a class="anchor" href="#pthreads"></a><a class="link" href="#pthreads">27.4.4. pthreads</a></h4>
<div class="paragraph">
<p>POSIX' multithreading API. Contrast with <a href="#fork">fork</a> which is for processes.</p>
</div>
<div class="paragraph">
<p>This was for a looong time the only "portable" multithreading alternative, until <a href="#cpp-multithreading">C++11 finally added threads</a>, thus also extending the portability to Windows.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_self.c">userland/posix/pthread_self.c</a>: the simplest example possible</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_count.c">userland/posix/pthread_count.c</a>: count an atomic varible across threads</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_deadlock.c">userland/posix/pthread_deadlock.c</a>: purposefully create a deadlock to see what it looks like</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_barrier.c">userland/posix/pthread_barrier.c</a>: related: <a href="https://stackoverflow.com/questions/28663622/understanding-posix-barrier-mechanism" class="bare">https://stackoverflow.com/questions/28663622/understanding-posix-barrier-mechanism</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="pthread-mutex"><a class="anchor" href="#pthread-mutex"></a><a class="link" href="#pthread-mutex">27.4.4.1. pthread_mutex</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/pthread_count.c">userland/posix/pthread_count.c</a> exemplifies the functions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>pthread_mutex_lock</code></p>
</li>
<li>
<p>pthread_mutex_unlock</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>That example that the same interface as: <a href="#atomic-cpp">atomic.cpp</a>.
There are no non-locking atomic types or atomic primitives in POSIX: <a href="http://stackoverflow.com/questions/1130018/unix-portable-atomic-operations" class="bare">http://stackoverflow.com/questions/1130018/unix-portable-atomic-operations</a></p>
</div>
<div class="paragraph">
<p><code>pthread_mutex_lock</code> and <code>pthread_mutex_unlock</code> and many other pthread functions already enforce cross thread memory synchronization:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/78172/using-c-pthreads-do-shared-variables-need-to-be-volatile/58935671#58935671" class="bare">https://stackoverflow.com/questions/78172/using-c-pthreads-do-shared-variables-need-to-be-volatile/58935671#58935671</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/3208060/does-guarding-a-variable-with-a-pthread-mutex-guarantee-its-also-not-cached" class="bare">https://stackoverflow.com/questions/3208060/does-guarding-a-variable-with-a-pthread-mutex-guarantee-its-also-not-cached</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/24137964/does-pthread-mutex-lock-contains-memory-fence-instruction" class="bare">https://stackoverflow.com/questions/24137964/does-pthread-mutex-lock-contains-memory-fence-instruction</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="sysconf"><a class="anchor" href="#sysconf"></a><a class="link" href="#sysconf">27.4.5. sysconf</a></h4>
<div class="paragraph">
<p><a href="https://pubs.opengroup.org/onlinepubs/9699919799/functions/sysconf.html" class="bare">https://pubs.opengroup.org/onlinepubs/9699919799/functions/sysconf.html</a></p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/sysconf.c">userland/posix/sysconf.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sysconf.c">userland/linux/sysconf.c</a> showcases Linux extensions to POSIX</p>
<div class="paragraph">
<p>Note that this blows up on gem5 userland due to <code>NPROCESSORS_ONLN</code> however: <a href="https://gem5.atlassian.net/browse/GEM5-622" class="bare">https://gem5.atlassian.net/browse/GEM5-622</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Get lots of info on the system configuration.</p>
</div>
<div class="paragraph">
<p>The constants can also be viewed accessed on my Ubuntu 18.04 host with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>getconf -a</pre>
</div>
</div>
<div class="paragraph">
<p><code>getconf</code> is also specified by POSIX at: <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/getconf.html" class="bare">https://pubs.opengroup.org/onlinepubs/9699919799/utilities/getconf.html</a> but not the <code>-a</code> option which shows all configurations.</p>
</div>
<div class="paragraph">
<p>Busybox 1.31.1 clearly states that <code>getconf</code> is not implemented however at <code>docs/posix_conformance.txt</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>POSIX Tools not supported:
  asa, at, batch, bc, c99, command, compress, csplit, ex, fc, file,
  gencat, getconf, iconv, join, link, locale, localedef, lp, m4,</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="mmap-2"><a class="anchor" href="#mmap-2"></a><a class="link" href="#mmap-2">27.4.6. mmap</a></h4>
<div class="paragraph">
<p>The mmap system call allows advanced memory operations.</p>
</div>
<div class="paragraph">
<p>mmap is notably used to implement the <a href="#malloc">malloc ANSI C</a> function, replacing the previously used break system call.</p>
</div>
<div class="paragraph">
<p>Linux adds has several POSIX extension flags to it.</p>
</div>
<div class="sect4">
<h5 id="mmap-map-anonymous"><a class="anchor" href="#mmap-map-anonymous"></a><a class="link" href="#mmap-map-anonymous">27.4.6.1. mmap MAP_ANONYMOUS</a></h5>
<div class="paragraph">
<p>Basic <code>mmap</code> example, do the same as <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/malloc.c">userland/c/malloc.c</a>, but with <code>mmap</code>.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/mmap_anonymous.c">userland/linux/mmap_anonymous.c</a></p>
</div>
<div class="paragraph">
<p>In POSIX 7 mmap always maps to a file.</p>
</div>
<div class="paragraph">
<p>If we add the MAP_ANONYMOUS Linux extension however, this is not required, and mmap can be used to allocate memory like malloc.</p>
</div>
<div class="paragraph">
<p>Answers: <a href="https://stackoverflow.com/questions/4779188/how-to-use-mmap-to-allocate-a-memory-in-heap" class="bare">https://stackoverflow.com/questions/4779188/how-to-use-mmap-to-allocate-a-memory-in-heap</a></p>
</div>
</div>
<div class="sect4">
<h5 id="mmap-file"><a class="anchor" href="#mmap-file"></a><a class="link" href="#mmap-file">27.4.6.2. mmap file</a></h5>
<div class="paragraph">
<p>Memory mapped file example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/mmap_file.c">userland/posix/mmap_file.c</a></p>
</div>
<div class="paragraph">
<p>The example creates a file, mmaps to it, writes to maped memory, and then closes the file.</p>
</div>
<div class="paragraph">
<p>We then read the file and confirm it was written to.</p>
</div>
</div>
<div class="sect4">
<h5 id="brk"><a class="anchor" href="#brk"></a><a class="link" href="#brk">27.4.6.3. brk</a></h5>
<div class="paragraph">
<p>Previously <a href="#posix">POSIX</a>, but was deprecated in favor of <a href="#malloc">malloc</a></p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/brk.c">userland/linux/brk.c</a></p>
</div>
<div class="paragraph">
<p>The example allocates two ints and uses them, and then deallocates back.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/6988487/what-does-the-brk-system-call-do/31082353#31082353" class="bare">https://stackoverflow.com/questions/6988487/what-does-the-brk-system-call-do/31082353#31082353</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="socket"><a class="anchor" href="#socket"></a><a class="link" href="#socket">27.4.7. socket</a></h4>
<div class="paragraph">
<p>A bit like <code>read</code> and <code>write</code>, but from / to the Internet!</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/wget.c">userland/posix/wget.c</a> tiny <code>wget</code> re-implementation. See: <a href="https://stackoverflow.com/questions/11208299/how-to-make-an-http-get-request-in-c-without-libcurl/35680609#35680609" class="bare">https://stackoverflow.com/questions/11208299/how-to-make-an-http-get-request-in-c-without-libcurl/35680609#35680609</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="userland-multithreading"><a class="anchor" href="#userland-multithreading"></a><a class="link" href="#userland-multithreading">27.5. Userland multithreading</a></h3>
<div class="paragraph">
<p>The following sections are related to multithreading in userland:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>language topics:</p>
<div class="ulist">
<ul>
<li>
<p><a href="#c-multithreading">C multithreading</a></p>
</li>
<li>
<p><a href="#cpp-multithreading">C++ multithreading</a></p>
</li>
<li>
<p><a href="#pthreads">pthreads</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>ISA topics:</p>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-thread-synchronization-primitives">x86 thread synchronization primitives</a></p>
</li>
<li>
<p><a href="#arm-thread-synchronization-primitives">ARM thread synchronization primitives</a></p>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-ldxr-and-stxr-instructions">ARM LDXR and STXR instructions</a></p>
</li>
<li>
<p><a href="#arm-lse">ARM Large System Extensions (LSE)</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>emulator topics:</p>
<div class="ulist">
<ul>
<li>
<p><a href="#qemu-user-mode-multithreading">QEMU user mode multithreading</a></p>
</li>
<li>
<p><a href="#gem5-syscall-emulation-multithreading">gem5 syscall emulation multithreading</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="c-debugging"><a class="anchor" href="#c-debugging"></a><a class="link" href="#c-debugging">27.6. C debugging</a></h3>
<div class="paragraph">
<p>Let&#8217;s group the hard-to-debug undefined-behaviour-like stuff found in C / C+ here and how to tackle those problems.</p>
</div>
<div class="sect3">
<h4 id="stack-smashing"><a class="anchor" href="#stack-smashing"></a><a class="link" href="#stack-smashing">27.6.1. Stack smashing</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264" class="bare">https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264</a></p>
</div>
<div class="dlist">
<dl>
<dt class="hdlist1">Example</dt>
<dd>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/smash_stack.c">userland/c/smash_stack.c</a></p>
</dd>
</dl>
</div>
<div class="paragraph">
<p>Leads to the dreadful "Stack smashing detected" message. Which is infinitely better than a silent break in any case.</p>
</div>
<div class="paragraph">
<p>We had also seen this error in our repository at: <a href="#stack-smashing-detected-when-using-glibc">stack smashing detected when using glibc</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="memory-leaks"><a class="anchor" href="#memory-leaks"></a><a class="link" href="#memory-leaks">27.6.2. Memory leaks</a></h4>
<div class="paragraph">
<p>How to debug: <a href="https://stackoverflow.com/questions/6261201/how-to-find-memory-leak-in-a-c-code-project/57877190#57877190" class="bare">https://stackoverflow.com/questions/6261201/how-to-find-memory-leak-in-a-c-code-project/57877190#57877190</a></p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/memory_leak.c">userland/c/memory_leak.c</a></p>
</div>
</div>
<div class="sect3">
<h4 id="profiling-userland-programs"><a class="anchor" href="#profiling-userland-programs"></a><a class="link" href="#profiling-userland-programs">27.6.3. Profiling userland programs</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/375913/how-can-i-profile-c-code-running-on-linux/60265409#60265409" class="bare">https://stackoverflow.com/questions/375913/how-can-i-profile-c-code-running-on-linux/60265409#60265409</a></p>
</div>
<div class="paragraph">
<p>OK, we have to learn this stuff.</p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/profile.c">userland/gcc/profile.c</a>: simple profiling example, where certain calls of a certain function can dominate the runtime</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="interpreted-languages"><a class="anchor" href="#interpreted-languages"></a><a class="link" href="#interpreted-languages">27.7. Interpreted languages</a></h3>
<div class="paragraph">
<p>Maybe some day someone will use this setup to study the performance of interpreters.</p>
</div>
<div class="sect3">
<h4 id="python"><a class="anchor" href="#python"></a><a class="link" href="#python">27.7.1. Python</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python">rootfs_overlay/lkmc/python</a></p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/hello.py">rootfs_overlay/lkmc/python/hello.py</a>: hello world</p>
</li>
<li>
<p><code>time</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/count.py">rootfs_overlay/lkmc/python/count.py</a>: count once every second</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/iter_method.py">rootfs_overlay/lkmc/python/iter_method.py</a>: how to implement <code><em>iter</em></code> on a class</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="python-standard-library"><a class="anchor" href="#python-standard-library"></a><a class="link" href="#python-standard-library">27.7.1.1. Python standard library</a></h5>
<div class="sect5">
<h6 id="python-unittest"><a class="anchor" href="#python-unittest"></a><a class="link" href="#python-unittest">27.7.1.1.1. Python unittest</a></h6>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/unittest_find/">rootfs_overlay/lkmc/python/unittest_find/</a> contains examples to test how tests are found by <code>unittest</code> within directories.  Related questions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/1732438/how-do-i-run-all-python-unit-tests-in-a-directory" class="bare">https://stackoverflow.com/questions/1732438/how-do-i-run-all-python-unit-tests-in-a-directory</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/46976256/recursive-unittest-discovery-with-python3-and-without-init-py-files" class="bare">https://stackoverflow.com/questions/46976256/recursive-unittest-discovery-with-python3-and-without-init-py-files</a></p>
</li>
</ul>
</div>
</div>
<div class="sect5">
<h6 id="python-relative-imports"><a class="anchor" href="#python-relative-imports"></a><a class="link" href="#python-relative-imports">27.7.1.1.2. Python relative imports</a></h6>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/relative_import/">rootfs_overlay/lkmc/python/relative_import/</a> contains examples to test how how to do relative imports in Python.</p>
</div>
<div class="paragraph">
<p>This subject is impossible to understand.</p>
</div>
<div class="paragraph">
<p>Related questions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/16981921/relative-imports-in-python-3" class="bare">https://stackoverflow.com/questions/16981921/relative-imports-in-python-3</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time" class="bare">https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/21490860/relative-imports-with-unittest-in-python" class="bare">https://stackoverflow.com/questions/21490860/relative-imports-with-unittest-in-python</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/714063/importing-modules-from-parent-folder" class="bare">https://stackoverflow.com/questions/714063/importing-modules-from-parent-folder</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect4">
<h5 id="build-and-install-the-interpreter"><a class="anchor" href="#build-and-install-the-interpreter"></a><a class="link" href="#build-and-install-the-interpreter">27.7.1.2. Build and install the interpreter</a></h5>
<div class="paragraph">
<p>Buildroot has a Python package that can be added to the guest image:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_PYTHON3=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Usage from guest in full system:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>and then from there get an interactive shell with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>python3</pre>
</div>
</div>
<div class="paragraph">
<p>or run an example with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>python3 python/hello.py</pre>
</div>
</div>
<div class="paragraph">
<p>or:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./python/hello.py</pre>
</div>
</div>
<div class="paragraph">
<p><a href="#user-mode-simulation">User mode simulation</a> interactive usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland "$(./getvar buildroot_target_dir)/usr/bin/python3"</pre>
</div>
</div>
<div class="paragraph">
<p>Non-interactive usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --userland "$(./getvar buildroot_target_dir)/usr/bin/python3" --cli-args rootfs_overlay/lkmc/python/hello.py</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="python-gem5-user-mode-simulation"><a class="anchor" href="#python-gem5-user-mode-simulation"></a><a class="link" href="#python-gem5-user-mode-simulation">27.7.1.3. Python gem5 user mode simulation</a></h5>
<div class="paragraph">
<p>At LKMC 50ac89b779363774325c81157ec8b9a6bdb50a2f gem5 390a74f59934b85d91489f8a563450d8321b602da:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --emulator gem5 \
  --userland "$(buildroot_target_dir)/usr/bin/python3" \
  --cli-args rootfs_overlay/lkmc/python/hello.py \
;</pre>
</div>
</div>
<div class="paragraph">
<p>fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fatal: Syscall 318 out of range</pre>
</div>
</div>
<div class="paragraph">
<p>which corresponds to the glorious <code>inotify_rm_watch</code> syscall: <a href="https://github.com/torvalds/linux/blob/v5.4/arch/arm/tools/syscall.tbl#L335" class="bare">https://github.com/torvalds/linux/blob/v5.4/arch/arm/tools/syscall.tbl#L335</a></p>
</div>
<div class="paragraph">
<p>and aarch64:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/usr/bin/python3" \
  --cli-args rootfs_overlay/lkmc/python/hello.py \
;</pre>
</div>
</div>
<div class="paragraph">
<p>fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fatal: syscall unused#278 (#278) unimplemented.</pre>
</div>
</div>
<div class="paragraph">
<p>which corresponds to the glorious <code>getrandom</code> syscall: <a href="https://github.com/torvalds/linux/blob/v4.17/include/uapi/asm-generic/unistd.h#L707" class="bare">https://github.com/torvalds/linux/blob/v4.17/include/uapi/asm-generic/unistd.h#L707</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/63999944/is-it-possible-to-run-python-code-in-gem5-syscall-emulation-mode" class="bare">https://stackoverflow.com/questions/63999944/is-it-possible-to-run-python-code-in-gem5-syscall-emulation-mode</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="embedding-python-in-another-application"><a class="anchor" href="#embedding-python-in-another-application"></a><a class="link" href="#embedding-python-in-another-application">27.7.1.4. Embedding Python in another application</a></h5>
<div class="paragraph">
<p>Here we will add some better examples and explanations for: <a href="https://docs.python.org/3/extending/embedding.html#very-high-level-embedding" class="bare">https://docs.python.org/3/extending/embedding.html#very-high-level-embedding</a></p>
</div>
<div class="paragraph">
<p>"Embedding Python" basically means calling the Python interpreter from C, and possibly passing values between the two.</p>
</div>
<div class="paragraph">
<p>These examples show to to embed the Python interpreter into a C/C++ application to interface between them</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/python_embed/eval.c">userland/libs/python_embed/eval.c</a>: this example simply does <code>eval</code> a Python string in C, and don&#8217;t communicate any values between the two.</p>
<div class="paragraph">
<p>It could be used to call external commands that have external side effects, but it is not very exciting.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/python_embed/pure.c">userland/libs/python_embed/pure.c</a>: this example actually defines some Python classes and functions from C, implementing those entirely in C.</p>
<div class="paragraph">
<p>The C program that defines those classes then instantiates the interpreter calls some regular Python code from it: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/python_embed/pure.py">userland/libs/python_embed/pure.py</a></p>
</div>
<div class="paragraph">
<p>The regular Python code can then use the native C classes as if they were defined in Python.</p>
</div>
<div class="paragraph">
<p>Finally, the Python returns values back to the C code that called the interpreter.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/python_embed/pure_cpp.cpp">userland/libs/python_embed/pure_cpp.cpp</a>: C version of the above, the main goal of this example is to show how to interface with C classes.</p>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/2200912/inheritance-in-python-c-extension/60436902#60436902" class="bare">https://stackoverflow.com/questions/2200912/inheritance-in-python-c-extension/60436902#60436902</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>One notable user of Python embedding is the <a href="#gem5">gem5</a> simulator, see also: <a href="#gem5-vs-qemu">gem5 vs QEMU</a>. gem5 embeds the Python interpreter in order to interpret scripts as seen from the CLI:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>build/ARM/gem5.opt configs/example/fs.py</pre>
</div>
</div>
<div class="paragraph">
<p>gem5 then runs that Python script, which instantiates C classes defined from Python, and then finally hands back control to the C runtime to run the actual simulation faster.</p>
</div>
</div>
<div class="sect4">
<h5 id="pybind11"><a class="anchor" href="#pybind11"></a><a class="link" href="#pybind11">27.7.1.5. pybind11</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/pybind11">userland/libs/pybind11</a></p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/145270/calling-c-c-from-python/60374990#60374990" class="bare">https://stackoverflow.com/questions/145270/calling-c-c-from-python/60374990#60374990</a></p>
</div>
<div class="paragraph">
<p>pybind11 is amazingly easy to use. But it can also make your builds really slow:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#pybind11-accounts-for-50-of-gem5-build-time">pybind11 accounts for 50% of gem5 build time</a>. As mentioned there, if pybind11 would split everything that can go into a cpp file from the hpp (i.e. everything except templates) that could already significantly reduce build times in certain cases. This is discussed upstream at: <a href="https://github.com/pybind/pybind11/issues/708" class="bare">https://github.com/pybind/pybind11/issues/708</a></p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/how-are-python-bindings-created/46453/2" class="bare">https://discuss.pytorch.org/t/how-are-python-bindings-created/46453/2</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="node-js"><a class="anchor" href="#node-js"></a><a class="link" href="#node-js">27.7.2. Node.js</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs">rootfs_overlay/lkmc/nodejs</a></p>
</div>
<div class="paragraph">
<p>Host installation shown at: <a href="https://askubuntu.com/questions/594656/how-to-install-the-latest-versions-of-nodejs-and-npm/971612#971612" class="bare">https://askubuntu.com/questions/594656/how-to-install-the-latest-versions-of-nodejs-and-npm/971612#971612</a></p>
</div>
<div class="paragraph">
<p>Build and install the interpreter in Buildroot with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_NODEJS=y'</pre>
</div>
</div>
<div class="paragraph">
<p>Everything is then the same as the <a href="#python">Python</a> interpreter setup, except that the executable name is now <code>node</code>!</p>
</div>
<div class="paragraph">
<p>TODO: build broken as of LKMC 3c3deb14dc8d6511680595dc42cb627d5781746d + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ERROR: package host-nodejs installs executables without proper RPATH</pre>
</div>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/hello.js">rootfs_overlay/lkmc/nodejs/hello.js</a>: hello world</p>
</li>
<li>
<p>String</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/alphanumeric.js">rootfs_overlay/lkmc/nodejs/alphanumeric.js</a>: <a href="https://stackoverflow.com/questions/4444477/how-to-tell-if-a-string-contains-a-certain-character-in-javascript/58359106#58359106" class="bare">https://stackoverflow.com/questions/4444477/how-to-tell-if-a-string-contains-a-certain-character-in-javascript/58359106#58359106</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>process</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/command_line_arguments.js">rootfs_overlay/lkmc/nodejs/command_line_arguments.js</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>fs</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/file_write_read.js">rootfs_overlay/lkmc/nodejs/file_write_read.js</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/read_stdin_to_string.js">rootfs_overlay/lkmc/nodejs/read_stdin_to_string.js</a> Question: <a href="https://stackoverflow.com/questions/30441025/read-all-text-from-stdin-to-a-string" class="bare">https://stackoverflow.com/questions/30441025/read-all-text-from-stdin-to-a-string</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>class</code></p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/object_to_string.js">rootfs_overlay/lkmc/nodejs/object_to_string.js</a>: <code>util.inspect.custom</code> and <code>toString</code> override experiment: <a href="https://stackoverflow.com/questions/24902061/is-there-an-repr-equivalent-for-javascript/26698403#26698403" class="bare">https://stackoverflow.com/questions/24902061/is-there-an-repr-equivalent-for-javascript/26698403#26698403</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/object_to_json.js">rootfs_overlay/lkmc/nodejs/object_to_json.js</a>: <code>toJSON</code> examples</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/static.js">rootfs_overlay/lkmc/nodejs/static.js</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/http.js">rootfs_overlay/lkmc/nodejs/http.js</a>: <code>http</code> module to create a simple HTTP server: <a href="https://nodejs.org/api/http.html" class="bare">https://nodejs.org/api/http.html</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/nodejs/esm">rootfs_overlay/lkmc/nodejs/esm</a>: <a href="https://stackoverflow.com/questions/58384179/syntaxerror-cannot-use-import-statement-outside-a-module" class="bare">https://stackoverflow.com/questions/58384179/syntaxerror-cannot-use-import-statement-outside-a-module</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="node-js-step-debugging"><a class="anchor" href="#node-js-step-debugging"></a><a class="link" href="#node-js-step-debugging">27.7.2.1. Node.js step debugging</a></h5>
<div class="paragraph">
<p>Overviews:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/12641679/nodejs-a-step-by-step-debugger-for-nodejs/60018317#60018317" class="bare">https://stackoverflow.com/questions/12641679/nodejs-a-step-by-step-debugger-for-nodejs/60018317#60018317</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/1911015/how-do-i-debug-node-js-applications/52423980#52423980" class="bare">https://stackoverflow.com/questions/1911015/how-do-i-debug-node-js-applications/52423980#52423980</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Skip breaking on the first line every time: <a href="https://stackoverflow.com/questions/41153179/why-is-the-node-debugger-break-on-first-line-a-thing" class="bare">https://stackoverflow.com/questions/41153179/why-is-the-node-debugger-break-on-first-line-a-thing</a></p>
</div>
<div class="paragraph">
<p>Break at function or line: <a href="https://stackoverflow.com/questions/65493221/how-to-break-at-a-specific-function-or-line-with-the-node-js-node-inspect-comman/65493318#65493318" class="bare">https://stackoverflow.com/questions/65493221/how-to-break-at-a-specific-function-or-line-with-the-node-js-node-inspect-comman/65493318#65493318</a></p>
</div>
<div class="paragraph">
<p>Show more context lines&#8230;&#8203; <a href="https://stackoverflow.com/questions/64942914/how-to-increase-the-number-of-context-lines-shown-in-the-node-js-debugger-when-u" class="bare">https://stackoverflow.com/questions/64942914/how-to-increase-the-number-of-context-lines-shown-in-the-node-js-debugger-when-u</a></p>
</div>
</div>
<div class="sect4">
<h5 id="npm"><a class="anchor" href="#npm"></a><a class="link" href="#npm">27.7.2.2. NPM</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Npm_(software" class="bare">https://en.wikipedia.org/wiki/Npm_(software</a>)</p>
</div>
<div class="paragraph">
<p>Some sample packages can be found under: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/npm">npm</a>.</p>
</div>
<div class="paragraph">
<p>Local testing of those packages can be done as shown at: <a href="https://stackoverflow.com/questions/59389027/how-to-interactively-test-the-executable-of-an-npm-node-js-package-during-develo" class="bare">https://stackoverflow.com/questions/59389027/how-to-interactively-test-the-executable-of-an-npm-node-js-package-during-develo</a></p>
</div>
<div class="paragraph">
<p>The packages will also be published to the NPM registry, so you can also play with them as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>npm install cirosantilli-&lt;directory-name&gt;</pre>
</div>
</div>
<div class="sect5">
<h6 id="npm-data-files"><a class="anchor" href="#npm-data-files"></a><a class="link" href="#npm-data-files">27.7.2.2.1. NPM data-files</a></h6>
<div class="paragraph">
<p>Illustrates how to add extra non-code data files to an NPM package, and then use those files at runtime.</p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/31642477/how-to-publish-a-npm-package-with-distribution-files/59407033#59407033" class="bare">https://stackoverflow.com/questions/31642477/how-to-publish-a-npm-package-with-distribution-files/59407033#59407033</a></p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="java"><a class="anchor" href="#java"></a><a class="link" href="#java">27.7.3. Java</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/java">rootfs_overlay/lkmc/java</a></p>
</div>
<div class="paragraph">
<p>No OpenJDK package as of 2018.08: <a href="https://stackoverflow.com/questions/28874150/buildroot-with-jamvm-2-0-for-java-8/59290927#59290927" class="bare">https://stackoverflow.com/questions/28874150/buildroot-with-jamvm-2-0-for-java-8/59290927#59290927</a> partly because their build system is shit like the rest of the project&#8217;s setup.</p>
</div>
<div class="paragraph">
<p>Unmerged patch at: <a href="http://lists.busybox.net/pipermail/buildroot/2018-February/213282.html" class="bare">http://lists.busybox.net/pipermail/buildroot/2018-February/213282.html</a></p>
</div>
<div class="paragraph">
<p>There is a JamVM package though <a href="https://en.wikipedia.org/wiki/JamVM" class="bare">https://en.wikipedia.org/wiki/JamVM</a> which is something Android started before moving to Dalvik,</p>
</div>
<div class="paragraph">
<p>Maybe some day other <a href="#android">Android</a> Java runtimes will also become compilable. Maybe, since Android is also shit.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="algorithms"><a class="anchor" href="#algorithms"></a><a class="link" href="#algorithms">27.8. Algorithms</a></h3>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm">userland/algorithm</a></p>
</div>
<div class="paragraph">
<p>This is still work in progress and needs better automation, but is already a good sketch. Key missing features:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>actually check that outputs are correct in <code>./test</code></p>
</li>
<li>
<p>create a mechanism to run all or some selected hand coded inputs</p>
</li>
<li>
<p>create a mechanism to run generated input</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The idea was originally started at: <a href="https://github.com/cirosantilli/algorithm-cheat" class="bare">https://github.com/cirosantilli/algorithm-cheat</a></p>
</div>
<div class="paragraph">
<p>The key idea is that input / output pairs are present in human readable files generated either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>manually for small test inputs</p>
</li>
<li>
<p>with a Python script for larger randomized tests</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Test programs then:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>read input from sdtin</p>
</li>
<li>
<p>produce output to stdout</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so that we can compare the output to the expected one.</p>
</div>
<div class="paragraph">
<p>This way, tests can be reused across several implementations in different languages, emulating the many multi-language programming competition websites out there.</p>
</div>
<div class="paragraph">
<p>For example, for a <a href="#userland-setup-getting-started-natively">native run</a> we can can run a set / sorting test:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd userland/algorithm/set
./build

# Run with a small hand written test.
./std_set.out &lt; test_data/8.i &gt; tmp.raw

# Extract the output from the sorted stdout, which also
# contained some timing information.
./parse_output output &lt; tmp.raw &gt; tmp.o

# Compare the output to the Expected one.
cmp tmp.o test_data/8.e

# Same but now with a large randomly generated input.
./generate_io
./std_set.out &lt; tmp.i | ./parse_output output &gt; tmp.o
cmp tmp.o tmp.e</pre>
</div>
</div>
<div class="paragraph">
<p>It is also possible to the algorithm tests normally from emulators in <a href="#user-mode-simulation">User mode simulation</a> by setting stdin as explained at <a href="#syscall-emulation-mode-program-stdin">syscall emulation mode program stdin</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 -u userland/algorithm/set/std_set.cpp --stdin-file userland/algorithm/set/test_data/8.i</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/generate_io">userland/algorithm/set/generate_io</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/main.hpp">userland/algorithm/set/main.hpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/parse_output">userland/algorithm/set/parse_output</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/std_set.cpp">userland/algorithm/set/std_set.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/test_data/8.e">userland/algorithm/set/test_data/8.e</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/test_data/8.i">userland/algorithm/set/test_data/8.i</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set/parse_output">userland/algorithm/set/parse_output</a> is needed because timing instrumentation measurements must be embedded in the program itself to allow:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>discounting the input reading / output writing operations from the actual "read / write to / from memory algorithm" itself</p>
</li>
<li>
<p>measuring the evolution of the benchmark mid way, e.g. to see how the current container size affects insertion time: <a href="#bst-vs-heap-vs-hashmap">BST vs heap vs hashmap</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The following are also interesting Buildroot libraries that we could benchmark:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Armadillo <code>C++</code>: linear algebra</p>
</li>
<li>
<p>fftw: Fourier transform</p>
</li>
<li>
<p>Flann</p>
</li>
<li>
<p>GSL: various</p>
</li>
<li>
<p>liblinear</p>
</li>
<li>
<p>libspacialindex</p>
</li>
<li>
<p>libtommath</p>
</li>
<li>
<p>qhull</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These are good targets for <a href="#gem5-run-benchmark">performance analysis with gem5</a>, and there is some overlap between this section and <a href="#benchmarks">Benchmarks</a>.</p>
</div>
<div class="sect3">
<h4 id="bst-vs-heap-vs-hashmap"><a class="anchor" href="#bst-vs-heap-vs-hashmap"></a><a class="link" href="#bst-vs-heap-vs-hashmap">27.8.1. BST vs heap vs hashmap</a></h4>
<div class="paragraph">
<p>TODO: move benchmark graph from <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/bst_vs_heap_vs_hashmap.cpp">userland/cpp/bst_vs_heap_vs_hashmap.cpp</a> to <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/algorithm/set">userland/algorithm/set</a>.</p>
</div>
<div class="paragraph">
<p>The following benchmark setup works both:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>on host through timers + <a href="https://stackoverflow.com/questions/51952471/why-do-i-get-a-constant-instead-of-logarithmic-curve-for-an-insert-time-benchmar/51953081#51953081">granule</a></p>
</li>
<li>
<p>gem5 with <a href="#m5ops-instructions">dumpstats</a>, which can get more precise results with <code>granule == 1</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It has been used to answer:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>BST vs heap: <a href="https://stackoverflow.com/questions/6147243/heap-vs-binary-search-tree-bst/29548834#29548834" class="bare">https://stackoverflow.com/questions/6147243/heap-vs-binary-search-tree-bst/29548834#29548834</a></p>
</li>
<li>
<p><code>std::set</code>: <a href="https://stackoverflow.com/questions/2558153/what-is-the-underlying-data-structure-of-a-stl-set-in-c/51944661#51944661" class="bare">https://stackoverflow.com/questions/2558153/what-is-the-underlying-data-structure-of-a-stl-set-in-c/51944661#51944661</a></p>
</li>
<li>
<p><code>std::map</code>: <a href="https://stackoverflow.com/questions/18414579/what-data-structure-is-inside-stdmap-in-c/51945119#51945119" class="bare">https://stackoverflow.com/questions/18414579/what-data-structure-is-inside-stdmap-in-c/51945119#51945119</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To benchmark on the host, we do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland-in-tree \
  --force-rebuild \
  --optimization-level 3 \
  ./userland/cpp/bst_vs_heap_vs_hashmap.cpp \
;
./userland/cpp/bst_vs_heap_vs_hashmap.out 10000000 10000 0 | tee bst_vs_heap_vs_hashmap.dat
gnuplot \
  -e 'input_noext="bst_vs_heap_vs_hashmap"' \
  -e 'heap_zoom_max=50' \
  -e 'hashmap_zoom_max=400' \
  ./bst-vs-heap-vs-hashmap.gnuplot \
;
xdg-open bst_vs_heap_vs_hashmap.tmp.png</pre>
</div>
</div>
<div class="paragraph">
<p>The parameters <code>heap_zoom_max</code> and <code>hashmap_zoom_max</code> are chosen manually interactively to best showcase the regions of interest in those plots.</p>
</div>
<div class="paragraph">
<p>To benchmark on gem5, we first build the benchmark with <a href="#m5ops-instructions">m5ops instructions</a> enabled, and then we run it and extract the stats:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland \
  --arch x86_64 \
  --ccflags='-DLKMC_M5OPS_ENABLE=1' \
  --force-rebuild userland/cpp/bst_vs_heap_vs_hashmap.cpp \
  --optimization-level 3 \
;
./run \
  --arch x86_64 \
  --emulator gem5 \
  --userland userland/cpp/bst_vs_heap_vs_hashmap.cpp \
  --cli-args='100000 1 0' \
  -- \
  --cpu-type=DerivO3CPU \
  --caches \
  --l2cache \
  --l1d_size=32kB \
  --l1i_size=32kB \
  --l2_size=256kB \
  --l3_size=20MB \
;
./bst-vs-heap-vs-hashmap-gem5-stats --arch x86_64 | tee bst_vs_heap_vs_hashmap_gem5.dat
gnuplot \
  -e 'input_noext="bst_vs_heap_vs_hashmap_gem5"' \
  -e 'heap_zoom_max=500' \
  -e 'hashmap_zoom_max=400' \
  ./bst-vs-heap-vs-hashmap.gnuplot \
;
xdg-open bst_vs_heap_vs_hashmap_gem5.tmp.png</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: the gem5 simulation blows up on a tcmalloc allocation somewhere near 25k elements as of 3fdd83c2c58327d9714fa2347c724b78d7c05e2b + 1, likely linked to the extreme inefficiency of the stats collection?</p>
</div>
<div class="paragraph">
<p>The cache sizes were chosen to match the host <a href="#p51">2017 Lenovo ThinkPad P51</a> to improve the comparison. Ideally we should also use the same standard library.</p>
</div>
<div class="paragraph">
<p>Note that this will take a long time, and will produce a humongous ~40Gb stats file as explained at: <a href="#gem5-only-dump-selected-stats">Section 24.10.3.2, &#8220;gem5 only dump selected stats&#8221;</a></p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/bst_vs_heap_vs_hashmap.cpp">userland/cpp/bst_vs_heap_vs_hashmap.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/bst-vs-heap-vs-hashmap-gem5-stats">bst-vs-heap-vs-hashmap-gem5-stats</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/bst-vs-heap-vs-hashmap.gnuplot">bst-vs-heap-vs-hashmap.gnuplot</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="blas"><a class="anchor" href="#blas"></a><a class="link" href="#blas">27.8.2. BLAS</a></h4>
<div class="paragraph">
<p>Buildroot supports it, which makes everything just trivial:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_OPENBLAS=y'
./build-userland --package openblas -- userland/libs/openblas/hello.c
./run --eval-after './libs/openblas/hello.out; echo $?'</pre>
</div>
</div>
<div class="paragraph">
<p>Outcome: the test passes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/openblas/hello.c">userland/libs/openblas/hello.c</a></p>
</div>
<div class="paragraph">
<p>The test performs a general matrix multiplication:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    |  1.0 -3.0 |   |  1.0  2.0  1.0 |       |  0.5  0.5  0.5 |   |  11.0 - 9.0  5.0 |
1 * |  2.0  4.0 | * | -3.0  4.0 -1.0 | + 2 * |  0.5  0.5  0.5 | = | - 9.0  21.0 -1.0 |
    |  1.0 -1.0 |                            |  0.5  0.5  0.5 |   |   5.0 - 1.0  3.0 |</pre>
</div>
</div>
<div class="paragraph">
<p>This can be deduced from the Fortran interfaces at</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar buildroot_build_build_dir)"/openblas-*/reference/dgemmf.f</pre>
</div>
</div>
<div class="paragraph">
<p>which we can map to our call as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>C := alpha*op( A )*op( B ) + beta*C,
SUBROUTINE DGEMMF(               TRANA,        TRANB,     M,N,K,  ALPHA,A,LDA,B,LDB,BETA,C,LDC)
cblas_dgemm(      CblasColMajor, CblasNoTrans, CblasTrans,3,3,2  ,1,    A,3,  B,3,  2   ,C,3  );</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="eigen"><a class="anchor" href="#eigen"></a><a class="link" href="#eigen">27.8.3. Eigen</a></h4>
<div class="paragraph">
<p>Header only linear algebra library with a mainline Buildroot package:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_EIGEN=y'
./build-userland --package eigen -- userland/libs/eigen/hello.cpp</pre>
</div>
</div>
<div class="paragraph">
<p>Just create an array and print it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './libs/eigen/hello.out'</pre>
</div>
</div>
<div class="paragraph">
<p>Output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>  3  -1
2.5 1.5</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs/eigen/hello.cpp">userland/libs/eigen/hello.cpp</a></p>
</div>
<div class="paragraph">
<p>This example just creates a matrix and prints it out.</p>
</div>
<div class="paragraph">
<p>Tested on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/a4bdcf102c068762bb1ef26c591fcf71e5907525">a4bdcf102c068762bb1ef26c591fcf71e5907525</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="benchmarks"><a class="anchor" href="#benchmarks"></a><a class="link" href="#benchmarks">27.9. Benchmarks</a></h3>
<div class="paragraph">
<p>These are good targets for <a href="#gem5-run-benchmark">performance analysis with gem5</a>.</p>
</div>
<div class="paragraph">
<p>TODO also consider the following:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://www.cs.virginia.edu/stream/ref.html" class="bare">http://www.cs.virginia.edu/stream/ref.html</a> STREAM memory bandwidth benchmarks.</p>
</li>
<li>
<p><a href="https://github.com/kozyraki/stamp" class="bare">https://github.com/kozyraki/stamp</a> transactional memory benchmarks</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="microbenchmarks"><a class="anchor" href="#microbenchmarks"></a><a class="link" href="#microbenchmarks">27.9.1. Microbenchmarks</a></h4>
<div class="paragraph">
<p>It eventually has to come to that, hasn&#8217;t it?</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> described at <a href="#c-busy-loop">C busy loop</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Of course, there is a continuum between what is a "microbenchmark" and a "macrobechmark".</p>
</div>
<div class="paragraph">
<p>One would hope that every microbenchmark exercises a concentrated subset of part of an important macro benchmark, otherwise what&#8217;s the point, right?</p>
</div>
<div class="paragraph">
<p>Also for parametrized "macro benchmark", you can always in theory reduce the problem size to be so small that it might be more appropriate to call it a micro benchmark.</p>
</div>
<div class="paragraph">
<p>So our working definition will be more of the type: "does it solve an understandable useful high level problem from start to end?".</p>
</div>
<div class="paragraph">
<p>If the answer is yes, then we call it a macro benchmark, otherwise micro.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/2842695/what-is-microbenchmarking" class="bare">https://stackoverflow.com/questions/2842695/what-is-microbenchmarking</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="userland-libs-directory"><a class="anchor" href="#userland-libs-directory"></a><a class="link" href="#userland-libs-directory">27.10. userland/libs directory</a></h3>
<div class="paragraph">
<p>Tests under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/libs">userland/libs</a> require certain optional libraries to be installed on the target, and are not built or tested by default, you must enable them with either:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>--package &lt;package&gt;
--package-all</pre>
</div>
</div>
<div class="paragraph">
<p>See for example <a href="#blas">BLAS</a>. Since it is located under <code>userland/libs/openblas</code>, it will only build with either:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --package openblas
./build-userland --package-all</pre>
</div>
</div>
<div class="paragraph">
<p>As an exception, if you first <code>cd</code> directly into one of the directories and do a <a href="#userland-setup-getting-started-natively">native host build</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt install libeigen3-dev
cd userland/libs/eigen
./build</pre>
</div>
</div>
<div class="paragraph">
<p>then that library will be automatically enabled.</p>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#cpp-third-party-libraries">C++ third-party libraries</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="userland-content-filename-conventions"><a class="anchor" href="#userland-content-filename-conventions"></a><a class="link" href="#userland-content-filename-conventions">27.11. Userland content filename conventions</a></h3>
<div class="paragraph">
<p>The following basenames should always refer to programs that do the same thing, but in different languages:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>count</code>: count to infinity, sleep one second between each number</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/count.sh">rootfs_overlay/lkmc/count.sh</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/python/count.py">rootfs_overlay/lkmc/python/count.py</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/count.cpp">userland/cpp/count.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/count.c">userland/posix/count.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="userland-content-bibliography"><a class="anchor" href="#userland-content-bibliography"></a><a class="link" href="#userland-content-bibliography">27.12. Userland content bibliography</a></h3>
<div class="ulist">
<ul>
<li>
<p>The Linux Programming Interface by Michael Kerrisk <a href="https://www.amazon.co.uk/Linux-Programming-Interface-System-Handbook/dp/1593272200" class="bare">https://www.amazon.co.uk/Linux-Programming-Interface-System-Handbook/dp/1593272200</a> Lots of open source POSIX examples: <a href="https://github.com/cirosantilli/linux-programming-interface-kerrisk" class="bare">https://github.com/cirosantilli/linux-programming-interface-kerrisk</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="userland-assembly"><a class="anchor" href="#userland-assembly"></a><a class="link" href="#userland-assembly">28. Userland assembly</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Programs under <code>userland/arch/&lt;arch&gt;/</code> are examples of userland assembly programming.</p>
</div>
<div class="paragraph">
<p>This section will document ISA agnostic concepts, and you should read it first.</p>
</div>
<div class="paragraph">
<p>ISA specifics are covered at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-userland-assembly">x86 userland assembly</a> under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/">userland/arch/x86_64/</a>, originally migrated from: <a href="https://github.com/cirosantilli/x86-assembly-cheat" class="bare">https://github.com/cirosantilli/x86-assembly-cheat</a></p>
</li>
<li>
<p><a href="#arm-userland-assembly">ARM userland assembly</a> originally migrated from <a href="https://github.com/cirosantilli/arm-assembly-cheat" class="bare">https://github.com/cirosantilli/arm-assembly-cheat</a> under:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/">userland/arch/arm/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/">userland/arch/aarch64/</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Like other userland programs, these programs can be run as explained at: <a href="#userland-setup">Section 2.8, &#8220;Userland setup&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>As a quick reminder, the fastest setups to get started are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#userland-setup-getting-started-natively">Userland setup getting started natively</a> if your host can run the examples, e.g. x86 example on an x86 host:</p>
</li>
<li>
<p><a href="#userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode">Userland setup getting started with prebuilt toolchain and QEMU user mode</a> otherwise</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>However, as usual, it is saner to build your toolchain as explained at: <a href="#qemu-user-mode-getting-started">Section 11.1, &#8220;QEMU user mode getting started&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The first examples you should look into are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>add</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/add.S">userland/arch/x86_64/add.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/add.S">userland/arch/arm/add.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/add.S">userland/arch/aarch64/add.S</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>mov between register and memory</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/mov.S">userland/arch/x86_64/mov.S</a></p>
</li>
<li>
<p><a href="#arm-mov-instruction">ARM MOV instruction</a></p>
</li>
<li>
<p><a href="#arm-load-and-store-instructions">ARM load and store instructions</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>addressing modes</p>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-addressing-modes">x86 addressing modes</a></p>
</li>
<li>
<p><a href="#arm-addressing-modes">ARM addressing modes</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>registers, see: <a href="#assembly-registers">Section 28.1, &#8220;Assembly registers&#8221;</a></p>
</li>
<li>
<p>jumping:</p>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-control-transfer-instructions">x86 control transfer instructions</a></p>
</li>
<li>
<p><a href="#arm-branch-instructions">ARM branch instructions</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>SIMD</p>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-simd">x86 SIMD</a></p>
</li>
<li>
<p><a href="#arm-simd">ARM SIMD</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The add examples in particular:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>introduce the basics of how a given assembly works: how many inputs / outputs, who is input and output, can it use memory or just registers, etc.</p>
<div class="paragraph">
<p>It is then a big copy paste for most other data instructions.</p>
</div>
</li>
<li>
<p>verify that the venerable ADD instruction and our assertions are working</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Now try to modify modify the x86_64 add program to see the assertion fail:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>LKMC_ASSERT_EQ(%rax, $4)</pre>
</div>
</div>
<div class="paragraph">
<p>because 1 + 2 tends to equal 3 instead of 4.</p>
</div>
<div class="paragraph">
<p>And then watch the assertion fail:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland
./run --userland userland/arch/x86_64/add.S</pre>
</div>
</div>
<div class="paragraph">
<p>with error message:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>assert_eq_64 failed
val1 0x3
val2 0x4
error: asm_main returned 1 at line 8</pre>
</div>
</div>
<div class="paragraph">
<p>and notice how the error message gives both:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the actual assembly source line number where the failing assert was</p>
</li>
<li>
<p>the actual and expected values</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Other infrastructure sanity checks that you might want to look into include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/empty.S">userland/arch/empty.S</a></p>
</li>
<li>
<p><code>LKMC_FAIL</code> tests</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/lkmc_assert_fail.S">userland/arch/lkmc_assert_fail.S</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>LKMC_ASSERT_EQ</code> tests</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/lkmc_assert_eq_fail.S">userland/arch/x86_64/lkmc_assert_eq_fail.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/lkmc_assert_eq_fail.S">userland/arch/arm/lkmc_assert_eq_fail.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/lkmc_assert_eq_fail.S">userland/arch/aarch64/lkmc_assert_eq_fail.S</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>LKMC_ASSERT_MEMCMP</code> tests</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/lkmc_assert_memcmp_fail.S">userland/arch/x86_64/lkmc_assert_memcmp_fail.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/lkmc_assert_memcmp_fail.S">userland/arch/arm/lkmc_assert_memcmp_fail.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/lkmc_assert_memcmp_fail.S">userland/arch/aarch64/lkmc_assert_memcmp_fail.S</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect2">
<h3 id="assembly-registers"><a class="anchor" href="#assembly-registers"></a><a class="link" href="#assembly-registers">28.1. Assembly registers</a></h3>
<div class="paragraph">
<p>After seeing an <a href="#userland-assembly">ADD hello world</a>, you need to learn the general registers:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86, see: <a href="#x86-registers">Section 29.1, &#8220;x86 registers&#8221;</a></p>
</li>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/registers.S">userland/arch/arm/registers.S</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/registers.S">userland/arch/aarch64/registers.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/pc.S">userland/arch/aarch64/pc.S</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="#armarm7">ARMv7 architecture reference manual</a> A2.3 "ARM core registers".</p>
</div>
<div class="sect3">
<h4 id="armv8-aarch64-x31-register"><a class="anchor" href="#armv8-aarch64-x31-register"></a><a class="link" href="#armv8-aarch64-x31-register">28.1.1. ARMv8 aarch64 x31 register</a></h4>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/x31.S">userland/arch/aarch64/x31.S</a></p>
</div>
<div class="paragraph">
<p>There is no X31 name, and the encoding can have two different names depending on the instruction:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>XZR: zero register:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/42788696/why-might-one-use-the-xzr-register-instead-of-the-literal-0-on-armv8" class="bare">https://stackoverflow.com/questions/42788696/why-might-one-use-the-xzr-register-instead-of-the-literal-0-on-armv8</a></p>
</li>
<li>
<p><a href="https://community.arm.com/processors/f/discussions/3185/wzr-xzr-register-s-purpose" class="bare">https://community.arm.com/processors/f/discussions/3185/wzr-xzr-register-s-purpose</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>SP: stack pointer</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To make things more confusing, some aliases can take either name, which makes them alias to different things, e.g. MOV accepts both:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov x0, sp
mov x0, xzr</pre>
</div>
</div>
<div class="paragraph">
<p>and the first one is an alias to ADD while the second an alias to <a href="#arm-bitwise-instructions">ORR</a>.</p>
</div>
<div class="paragraph">
<p>The difference is documented on a per instruction basis. Instructions that encode 31 as SP say:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if d == 31 then
  SP[] = result;
else
  X[d] = result;</pre>
</div>
</div>
<div class="paragraph">
<p>And then those that don&#8217;t say that, B1.2.1 "Registers in AArch64 state" implies the zero register:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>In instruction encodings, the value 0b11111 (31) is used to indicate the ZR (zero register). This
indicates that the argument takes the value zero, but does not indicate that the ZR is implemented
as a physical register.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>This is also described on <a href="#armarm8">ARMv8 architecture reference manual</a> C1.2.5 "Register names":</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>There is no register named W31 or X31.</p>
</div>
<div class="paragraph">
<p>The name SP represents the stack pointer for 64-bit operands where an encoding of the value 31 in the
corresponding register field is interpreted as a read or write of the current stack pointer. When instructions
do not interpret this operand encoding as the stack pointer, use of the name SP is an error.</p>
</div>
<div class="paragraph">
<p>The name XZR represents the zero register for 64-bit operands where an encoding of the value 31 in the
corresponding register field is interpreted as returning zero when read or discarding the result when written.
When instructions do not interpret this operand encoding as the zero register, use of the name XZR is an error</p>
</div>
</blockquote>
</div>
</div>
</div>
<div class="sect2">
<h3 id="floating-point-assembly"><a class="anchor" href="#floating-point-assembly"></a><a class="link" href="#floating-point-assembly">28.2. Floating point assembly</a></h3>
<div class="paragraph">
<p>Keep in mind that many ISAs started floating point as an optional thing, and it later got better integrated into the main CPU, side by side with SIMD.</p>
</div>
<div class="paragraph">
<p>For this reason, there are sometimes multiple ways to do floating point operations in each ISA.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s start as usual with floating point addition + register file:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-vadd-instruction">ARM VADD instruction</a></p>
</li>
<li>
<p><a href="#arm-vfp-registers">ARM VFP registers</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="#armv8-aarch64-fadd-instruction">ARMv8 aarch64 FADD instruction</a></p>
</li>
<li>
<p><a href="#armv8-aarch64-floating-point-registers">ARMv8 AArch64 floating point registers</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="simd-assembly"><a class="anchor" href="#simd-assembly"></a><a class="link" href="#simd-assembly">28.3. SIMD assembly</a></h3>
<div class="paragraph">
<p>Much like ADD for non-SIMD, start learning SIMD instructions by looking at the integer and floating point SIMD ADD instructions of each ISA:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86</p>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-sse-data-transfer-instructions">ADDPD</a></p>
</li>
<li>
<p><a href="#x86-paddq-instruction">x86 PADDQ instruction</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-vadd-instruction">ARM VADD instruction</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="#armv8-aarch64-add-vector-instruction">ARMv8 aarch64 add vector instruction</a></p>
</li>
<li>
<p><a href="#armv8-aarch64-fadd-instruction">ARMv8 aarch64 FADD instruction</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Then it is just a huge copy paste of infinite boring details:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-simd">x86 SIMD</a></p>
</li>
<li>
<p><a href="#arm-simd">ARM SIMD</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To debug these instructions, you can see the register values in GDB with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info registers float</pre>
</div>
</div>
<div class="paragraph">
<p>or alternatively with register names (here the ARMv8 V0 register):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>print $v0</pre>
</div>
</div>
<div class="paragraph">
<p>as mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/5429137/how-to-print-register-values-in-gdb/38036152#38036152" class="bare">https://stackoverflow.com/questions/5429137/how-to-print-register-values-in-gdb/38036152#38036152</a></p>
</li>
<li>
<p><a href="https://reverseengineering.stackexchange.com/questions/8992/floating-point-registers-on-arm/20623#20623" class="bare">https://reverseengineering.stackexchange.com/questions/8992/floating-point-registers-on-arm/20623#20623</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/1389712/getting-started-with-intel-x86-sse-simd-instructions/56409539#56409539" class="bare">https://stackoverflow.com/questions/1389712/getting-started-with-intel-x86-sse-simd-instructions/56409539#56409539</a></p>
</div>
<div class="sect3">
<h4 id="fma-instruction"><a class="anchor" href="#fma-instruction"></a><a class="link" href="#fma-instruction">28.3.1. FMA instruction</a></h4>
<div class="paragraph">
<p>Fused multiply add:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86: <a href="#x86-fma">Section 29.12.3, &#8220;x86 fused multiply add (FMA)&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/Multiply–accumulate_operation" class="bare">https://en.wikipedia.org/wiki/Multiply–accumulate_operation</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/FMA_instruction_set" class="bare">https://en.wikipedia.org/wiki/FMA_instruction_set</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Particularly important numerical analysis instruction, that is used in particular for;</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Dot product</p>
</li>
<li>
<p>Matrix multiplication</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>FMA is so important that <a href="#ieee-754">IEEE 754</a> specifies it with single precision drop compared to a separate add and multiply!</p>
</div>
<div class="paragraph">
<p>Micro-op fun: <a href="https://stackoverflow.com/questions/28630864/how-is-fma-implemented" class="bare">https://stackoverflow.com/questions/28630864/how-is-fma-implemented</a></p>
</div>
<div class="paragraph">
<p>Historically, FMA instructions have been added relatively late to instruction sets.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="user-vs-system-assembly"><a class="anchor" href="#user-vs-system-assembly"></a><a class="link" href="#user-vs-system-assembly">28.4. User vs system assembly</a></h3>
<div class="paragraph">
<p>By "userland assembly", we mean "the parts of the ISA which can be freely used from userland".</p>
</div>
<div class="paragraph">
<p>Most ISAs are divided into a system and userland part, and to running the system part requires elevated privileges such as <a href="#ring0">ring0</a> in x86.</p>
</div>
<div class="paragraph">
<p>One big difference between both is that we can run userland assembly on <a href="#userland-setup">Userland setup</a>, which is easier to get running and debug.</p>
</div>
<div class="paragraph">
<p>In particular, most userland assembly examples link to the C standard library, see: <a href="#userland-assembly-c-standard-library">Section 28.5, &#8220;Userland assembly C standard library&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Userland assembly is generally simpler, and a pre-requisite for <a href="#baremetal-setup">Baremetal setup</a>.</p>
</div>
<div class="paragraph">
<p>System-land assembly cheats will be put under: <a href="#baremetal-setup">Section 2.9, &#8220;Baremetal setup&#8221;</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="userland-assembly-c-standard-library"><a class="anchor" href="#userland-assembly-c-standard-library"></a><a class="link" href="#userland-assembly-c-standard-library">28.5. Userland assembly C standard library</a></h3>
<div class="paragraph">
<p>All examples except the <a href="#freestanding-programs">Freestanding programs</a> link to the C standard library.</p>
</div>
<div class="paragraph">
<p>This allows using the C standard library for IO, which is very convenient and portable across host OSes.</p>
</div>
<div class="paragraph">
<p>It also exposes other non-IO functionality that is very convenient such as <code>memcmp</code>.</p>
</div>
<div class="paragraph">
<p>The C standard library infrastructure is implemented in the common userland / baremetal source files:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc.c">lkmc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc.h">lkmc.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/aarch64.h">lkmc/aarch64.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/arm.h">lkmc/arm.h</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/x86_64.h">lkmc/x86_64.h</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="freestanding-programs"><a class="anchor" href="#freestanding-programs"></a><a class="link" href="#freestanding-programs">28.5.1. Freestanding programs</a></h4>
<div class="paragraph">
<p>Unlike most our other assembly examples, which use the C standard library for portability, examples under <code>freestanding/</code> directories don&#8217;t link to the C standard library:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/">userland/freestanding/</a>: freestanding programs that work on any ISA</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/freestanding/">userland/arch/x86_64/freestanding/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/freestanding/">userland/arch/arm/freestanding/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/">userland/arch/aarch64/freestanding/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>As a result, those examples cannot do IO portably, and so they make raw system calls and only be run on one given OS, e.g. <a href="#linux-system-calls">Linux system calls</a>.</p>
</div>
<div class="paragraph">
<p>Such executables are called freestanding because they don&#8217;t execute the glibc initialization code, but rather start directly on our custom hand written assembly.</p>
</div>
<div class="paragraph">
<p>In order to GDB step debug those executables, you will want to use <code>--no-continue</code>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --userland userland/arch/aarch64/freestanding/linux/hello.S --gdb-wait
./run-gdb --arch aarch64 --no-continue --userland userland/arch/aarch64/freestanding/linux/hello.S</pre>
</div>
</div>
<div class="paragraph">
<p>or in one go with <a href="#tmux">tmux</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --gdb-wait \
  --tmux-args=--no-continue \
  --userland userland/arch/aarch64/freestanding/linux/hello.S \
;</pre>
</div>
</div>
<div class="paragraph">
<p>You are now left on the very first instruction of our tiny executable!</p>
</div>
<div class="paragraph">
<p>This is analogous to <a href="#baremetal-gdb-step-debug">step debugging baremetal examples</a>.</p>
</div>
<div class="paragraph">
<p>Related:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/4783404/is-main-really-start-of-a-c-program/64116561#64116561" class="bare">https://stackoverflow.com/questions/4783404/is-main-really-start-of-a-c-program/64116561#64116561</a> "Is main() really start of a C++ program?"</p>
</li>
<li>
<p><a href="https://electronics.stackexchange.com/questions/258896/what-happens-before-main/404298#404298" class="bare">https://electronics.stackexchange.com/questions/258896/what-happens-before-main/404298#404298</a></p>
</li>
<li>
<p><a href="https://electronics.stackexchange.com/questions/55767/who-receives-the-value-returned-by-main" class="bare">https://electronics.stackexchange.com/questions/55767/who-receives-the-value-returned-by-main</a>, more microcontroller focused, should entitled "how to quit a program in microcontroller"</p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/53570678/what-happens-before-main-in-c" class="bare">https://stackoverflow.com/questions/53570678/what-happens-before-main-in-c</a> "What happens before main in C++?"</p>
</li>
<li>
<p><a href="https://www.quora.com/What-is-happening-before-the-main-function-is-called-in-C++-programming" class="bare">https://www.quora.com/What-is-happening-before-the-main-function-is-called-in-C++-programming</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/2053029/how-exactly-does-attribute-constructor-work" class="bare">https://stackoverflow.com/questions/2053029/how-exactly-does-attribute-constructor-work</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="nostartfiles-programs"><a class="anchor" href="#nostartfiles-programs"></a><a class="link" href="#nostartfiles-programs">28.5.1.1. nostartfiles programs</a></h5>
<div class="paragraph">
<p>Assembly examples under <code>nostartfiles</code> directories can use the standard library, but they don&#8217;t use the pre-<code>main</code> boilerplate and start directly at our explicitly given <code>_start</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/nostartfiles/">userland/arch/x86_64/nostartfiles/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/nostartfiles/">userland/arch/aarch64/nostartfiles/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>I&#8217;m not sure how much stdlib functionality is supposed to work without the pre-main stuff, but I guess we&#8217;ll just have to find out!</p>
</div>
<div class="paragraph">
<p>Was going to ask the following markdown question, but I noticed half way that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>without <code>-static</code>, I see a bunch of dynamic loader instructions, so not much is gained</p>
</li>
<li>
<p>with <code>-static</code>, the program segfaults, including on the host with stack:</p>
<div class="literalblock">
<div class="content">
<pre>#0  0x0000000000429625 in _IO_cleanup ()
#1  0x0000000000400c72 in __run_exit_handlers ()
#2  0x0000000000400caa in exit ()
#3  0x0000000000400a01 in _start () at exit.S:4</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>so I didn&#8217;t really have a good question.</p>
</div>
<div class="paragraph">
<p>The Markdown question that was almost asked:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>When working in emulators, I often want to keep my workloads as small as possible to more easily study instruction traces and reproduce bugs.

One of the ways I often want to do that, especially when doing [user mode simulations](https://wiki.debian.org/QemuUserEmulation), is by not running [the code that normally runs before main](https://stackoverflow.com/questions/53570678/what-happens-before-main-in-c) so that I can start directly in the instructions of interest that I control myself, which can be achieved with the `gcc -nostartfiles` option and by starting the program directly at `_start`.

Here is a tiny example that calls just `exit` from the C standard library:

main.S</pre>
</div>
</div>
<div class="paragraph">
<div class="title">global _start</div>
<p>_start:
    mov $0, %rdi
    call exit</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Compile and run with:</pre>
</div>
</div>
<div class="paragraph">
<p>gcc -ggdb3 -nostartfiles -static -o exit.out exit.S
qemu-x86_64 -d in_asm exit.out</p>
</div>
<div class="literalblock">
<div class="content">
<pre>However, for programming convenience, and to potentially keep my examples more OS portable, I would like to avoid making raw system calls, which would of course work, by using C standard library functions instead.

But I'm afraid that some of those C standard library functions will fail in subtle ways because I have skipped required initialization steps that would normally happen before `main`.

Is it any easy to determine which functions I can use or not, in case there are any that I can't use?</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gcc-inline-assembly"><a class="anchor" href="#gcc-inline-assembly"></a><a class="link" href="#gcc-inline-assembly">28.6. GCC inline assembly</a></h3>
<div class="paragraph">
<p>Examples under <code>arch/&lt;arch&gt;/c/</code> directories show to how use inline assembly from higher level languages such as C:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/inc.c">userland/arch/x86_64/inline_asm/inc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/add.c">userland/arch/x86_64/inline_asm/add.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/sqrt_x87.c">userland/arch/x86_64/inline_asm/sqrt_x87.c</a> Shows how to use the <a href="#x86-x87-fpu-instructions">x86 x87 FPU instructions</a> from inline assembly. Bibliography: <a href="https://stackoverflow.com/questions/6514537/how-do-i-specify-immediate-floating-point-numbers-with-inline-assembly/52906126#52906126" class="bare">https://stackoverflow.com/questions/6514537/how-do-i-specify-immediate-floating-point-numbers-with-inline-assembly/52906126#52906126</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/inc.c">userland/arch/arm/inline_asm/inc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/inc_memory.c">userland/arch/arm/inline_asm/inc_memory.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/inc_memory_global.c">userland/arch/arm/inline_asm/inc_memory_global.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/add.c">userland/arch/arm/inline_asm/add.c</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/earlyclobber.c">userland/arch/aarch64/inline_asm/earlyclobber.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/inc.c">userland/arch/aarch64/inline_asm/inc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/inc_32.c">userland/arch/aarch64/inline_asm/inc_32.c</a>: how to use 32-bit <code>w</code> registers in aarch64. We have to add <code>w</code> to the <code>%</code> as in <code>%w[io]</code> instead of <code>%[io]</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/multiline.cpp">userland/arch/aarch64/inline_asm/multiline.cpp</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gcc-inline-assembly-register-variables"><a class="anchor" href="#gcc-inline-assembly-register-variables"></a><a class="link" href="#gcc-inline-assembly-register-variables">28.6.1. GCC inline assembly register variables</a></h4>
<div class="paragraph">
<p>Used notably in some of the <a href="#linux-system-calls">Linux system calls</a> setups:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/reg_var.c">userland/arch/arm/inline_asm/reg_var.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/reg_var.c">userland/arch/aarch64/inline_asm/reg_var.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/reg_var_float.c">userland/arch/aarch64/inline_asm/reg_var_float.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In x86, makes it possible to access variables not exposed with the one letter register constraints.</p>
</div>
<div class="paragraph">
<p>In arm, it is the only way to achieve this effect: <a href="https://stackoverflow.com/questions/10831792/how-to-use-specific-register-in-arm-inline-assembler" class="bare">https://stackoverflow.com/questions/10831792/how-to-use-specific-register-in-arm-inline-assembler</a></p>
</div>
<div class="paragraph">
<p>This feature notably useful for making system calls from C, see: <a href="#linux-system-calls">Section 28.7, &#8220;Linux system calls&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Documentation: <a href="https://gcc.gnu.org/onlinedocs/gcc-4.4.2/gcc/Explicit-Reg-Vars.html" class="bare">https://gcc.gnu.org/onlinedocs/gcc-4.4.2/gcc/Explicit-Reg-Vars.html</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gcc-inline-assembly-scratch-registers"><a class="anchor" href="#gcc-inline-assembly-scratch-registers"></a><a class="link" href="#gcc-inline-assembly-scratch-registers">28.6.2. GCC inline assembly scratch registers</a></h4>
<div class="paragraph">
<p>How to use temporary registers in inline assembly:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/scratch.c">userland/arch/x86_64/inline_asm/scratch.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/scratch_hardcode.c">userland/arch/x86_64/inline_asm/scratch_hardcode.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/6682733/gcc-prohibit-use-of-some-registers/54963829#54963829" class="bare">https://stackoverflow.com/questions/6682733/gcc-prohibit-use-of-some-registers/54963829#54963829</a></p>
</div>
</div>
<div class="sect3">
<h4 id="gcc-inline-assembly-early-clobbers"><a class="anchor" href="#gcc-inline-assembly-early-clobbers"></a><a class="link" href="#gcc-inline-assembly-early-clobbers">28.6.3. GCC inline assembly early-clobbers</a></h4>
<div class="paragraph">
<p>An example of using the <code>&amp;</code> early-clobber modifier: link:userland/arch/aarch64/earlyclobber.c</p>
</div>
<div class="paragraph">
<p>More details at: <a href="https://stackoverflow.com/questions/15819794/when-to-use-earlyclobber-constraint-in-extended-gcc-inline-assembly/54853663#54853663" class="bare">https://stackoverflow.com/questions/15819794/when-to-use-earlyclobber-constraint-in-extended-gcc-inline-assembly/54853663#54853663</a></p>
</div>
<div class="paragraph">
<p>The assertion may fail without it. It actually does fail in GCC 8.2.0.</p>
</div>
</div>
<div class="sect3">
<h4 id="gcc-inline-assembly-floating-point-arm"><a class="anchor" href="#gcc-inline-assembly-floating-point-arm"></a><a class="link" href="#gcc-inline-assembly-floating-point-arm">28.6.4. GCC inline assembly floating point ARM</a></h4>
<div class="paragraph">
<p>Not documented as of GCC 8.2, but possible: <a href="https://stackoverflow.com/questions/53960240/armv8-floating-point-output-inline-assembly" class="bare">https://stackoverflow.com/questions/53960240/armv8-floating-point-output-inline-assembly</a></p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/inc_float.c">userland/arch/arm/inline_asm/inc_float.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/inc_float.c">userland/arch/aarch64/inline_asm/inc_float.c</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gcc-intrinsics"><a class="anchor" href="#gcc-intrinsics"></a><a class="link" href="#gcc-intrinsics">28.6.5. GCC intrinsics</a></h4>
<div class="paragraph">
<p>Pre-existing C wrappers using inline assembly, this is what production programs should use instead of inline assembly for SIMD:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/intrinsics/paddq.c">userland/arch/x86_64/intrinsics/paddq.c</a>. Intrinsics version of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/paddq.S">userland/arch/x86_64/paddq.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/intrinsics/addpd.c">userland/arch/x86_64/intrinsics/addpd.c</a>. Intrinsics version of <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/addpd.S">userland/arch/x86_64/addpd.S</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="gcc-x86-intrinsics"><a class="anchor" href="#gcc-x86-intrinsics"></a><a class="link" href="#gcc-x86-intrinsics">28.6.5.1. GCC x86 intrinsics</a></h5>
<div class="paragraph">
<p>Good official cheatsheet with all intrinsics and what they expand to: <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide" class="bare">https://software.intel.com/sites/landingpage/IntrinsicsGuide</a></p>
</div>
<div class="paragraph">
<p>The functions use the the following naming convention:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;vector_size&gt;_&lt;intrin_op&gt;_&lt;suffix&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>&lt;vector_size&gt;</code>:</p>
<div class="ulist">
<ul>
<li>
<p><code>mm</code>: 128-bit vectors (SSE)</p>
</li>
<li>
<p><code>mm256</code>: 256-bit vectors (AVX and AVX2)</p>
</li>
<li>
<p><code>mm512</code>: 512-bit vectors (AVX512)</p>
</li>
</ul>
</div>
</li>
<li>
<p><code>&lt;intrin_op&gt;</code>: operation of the intrinsic function, e.g. add, sub, mul, etc.</p>
</li>
<li>
<p><code>&lt;suffix&gt;</code>: data type:</p>
<div class="ulist">
<ul>
<li>
<p><code>ps</code>: 4 floats (Packed Single)</p>
</li>
<li>
<p><code>pd</code>: 2 doubles (Packed Double)</p>
</li>
<li>
<p><code>ss</code>: 1 float (Single Single), often the lowest order one</p>
</li>
<li>
<p><code>sd</code>: 1 double (Single Double)</p>
</li>
<li>
<p><code>si128</code>: 128-bits of integers of any size</p>
</li>
<li>
<p><code>ep&lt;int_type&gt;</code> integer types, e.g.:</p>
<div class="ulist">
<ul>
<li>
<p><code>epi32</code>: 32 bit signed integers</p>
</li>
<li>
<p><code>epu16</code>: 16 bit unsigned integers</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Data types:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>__m128</code>: four floats</p>
</li>
<li>
<p><code>__m128d</code>: two doubles</p>
</li>
<li>
<p><code>__m128i</code>: integers: 8 x 16-bit, 4 x 32-bit, 2 x 64-bit</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The headers to include are clarified at: <a href="https://stackoverflow.com/questions/11228855/header-files-for-x86-simd-intrinsics" class="bare">https://stackoverflow.com/questions/11228855/header-files-for-x86-simd-intrinsics</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>x86intrin.h everything
mmintrin.h  MMX
xmmintrin.h SSE
emmintrin.h SSE2
pmmintrin.h SSE3
tmmintrin.h SSSE3
smmintrin.h SSE4.1
nmmintrin.h SSE4.2
ammintrin.h SSE4A
wmmintrin.h AES
immintrin.h AVX
zmmintrin.h AVX512</pre>
</div>
</div>
<div class="paragraph">
<p>Present in <code>gcc-7_3_0-release</code> tree at: <code>gcc/config/i386/x86intrin.h</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.cs.virginia.edu/~cr4bd/3330/S2018/simdref.html" class="bare">https://www.cs.virginia.edu/~cr4bd/3330/S2018/simdref.html</a></p>
</li>
<li>
<p><a href="https://software.intel.com/en-us/articles/how-to-use-intrinsics" class="bare">https://software.intel.com/en-us/articles/how-to-use-intrinsics</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-system-calls"><a class="anchor" href="#linux-system-calls"></a><a class="link" href="#linux-system-calls">28.7. Linux system calls</a></h3>
<div class="paragraph">
<p>The following <a href="#userland-setup">Userland setup</a> programs illustrate how to make system calls:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/freestanding/linux/hello.S">userland/arch/x86_64/freestanding/linux/hello.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/freestanding/linux/int_system_call.S">userland/arch/x86_64/freestanding/linux/int_system_call.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/freestanding/linux/hello.c">userland/arch/x86_64/inline_asm/freestanding/linux/hello.c</a>: this shows how to do system calls from inline assembly without any C standard library helpers like <code>syscall</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/freestanding/linux/hello_regvar.c">userland/arch/x86_64/inline_asm/freestanding/linux/hello_regvar.c</a>: same as <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inline_asm/freestanding/linux/hello.c">userland/arch/x86_64/inline_asm/freestanding/linux/hello.c</a> but using register variables instead of register constraints</p>
</li>
</ul>
</div>
</li>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/freestanding/linux/hello.S">userland/arch/arm/freestanding/linux/hello.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inline_asm/freestanding/linux/hello.c">userland/arch/arm/inline_asm/freestanding/linux/hello.c</a>: there are no register constraints in ARM, so register variables are the most efficient way of storing variables in specific general purpose registers: <a href="https://stackoverflow.com/questions/3929442/how-to-specify-an-individual-register-as-constraint-in-arm-gcc-inline-assembly/54845046#54845046" class="bare">https://stackoverflow.com/questions/3929442/how-to-specify-an-individual-register-as-constraint-in-arm-gcc-inline-assembly/54845046#54845046</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/hello.S">userland/arch/aarch64/freestanding/linux/hello.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/freestanding/linux/hello.c">userland/arch/aarch64/inline_asm/freestanding/linux/hello.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/freestanding/linux/hello_clobbers.c">userland/arch/aarch64/inline_asm/freestanding/linux/hello_clobbers.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Determining the ARM syscall numbers:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://reverseengineering.stackexchange.com/questions/16917/arm64-syscalls-table" class="bare">https://reverseengineering.stackexchange.com/questions/16917/arm64-syscalls-table</a></p>
</li>
<li>
<p>arm: <a href="https://github.com/torvalds/linux/blob/v4.17/arch/arm/tools/syscall.tbl" class="bare">https://github.com/torvalds/linux/blob/v4.17/arch/arm/tools/syscall.tbl</a></p>
</li>
<li>
<p>aarch64: <a href="https://github.com/torvalds/linux/blob/v4.17/include/uapi/asm-generic/unistd.h" class="bare">https://github.com/torvalds/linux/blob/v4.17/include/uapi/asm-generic/unistd.h</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Determining the ARM syscall interface:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/12946958/what-is-the-interface-for-arm-system-calls-and-where-is-it-defined-in-the-linux" class="bare">https://stackoverflow.com/questions/12946958/what-is-the-interface-for-arm-system-calls-and-where-is-it-defined-in-the-linux</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/45742869/linux-syscall-conventions-for-armv8" class="bare">https://stackoverflow.com/questions/45742869/linux-syscall-conventions-for-armv8</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Questions about the C inline assembly examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86_64: <a href="https://stackoverflow.com/questions/9506353/how-to-invoke-a-system-call-via-sysenter-in-inline-assembly/54956854#54956854" class="bare">https://stackoverflow.com/questions/9506353/how-to-invoke-a-system-call-via-sysenter-in-inline-assembly/54956854#54956854</a></p>
</li>
<li>
<p>ARM: <a href="https://stackoverflow.com/questions/21729497/doing-a-syscall-without-libc-using-arm-inline-assembly" class="bare">https://stackoverflow.com/questions/21729497/doing-a-syscall-without-libc-using-arm-inline-assembly</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="futex-system-call"><a class="anchor" href="#futex-system-call"></a><a class="link" href="#futex-system-call">28.7.1. futex system call</a></h4>
<div class="paragraph">
<p>This is how threads either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>request the kernel to sleep until they are woken up by other threads</p>
</li>
<li>
<p>request the kernel to wake up other threads that are waiting on a given futex</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This syscall is rarely used on its own, and there isn&#8217;t even a glibc wrapper for it: you almost always just want to use the <a href="#pthreads">pthreads</a> or <a href="#cpp-multithreading">C++ multithreading</a> wrappers which use it for you to <a href="#userland-mutex-implementation">implement higher level constructs like mutexes</a>.</p>
</div>
<div class="paragraph">
<p>Futexes are bit complicated, because in order to achieve their efficiency, basically nothing is guaranteed: the wait might not wait, and the wakes might not wake.</p>
</div>
<div class="paragraph">
<p>So you are just basically forced to use atomic operations on the futex memory address in order to be sure of anything (we encourage you to try without :-)).</p>
</div>
<div class="paragraph">
<p>Minimal examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/futex.h">lkmc/futex.h</a>: our futex wrapper</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/futex.c">userland/linux/futex.c</a>: minimal example. It:</p>
<div class="ulist">
<ul>
<li>
<p>first spawns a child</p>
</li>
<li>
<p>then sleeps for 1 second and wakes up the futex if anyone is sleeping on it</p>
</li>
<li>
<p>the child sleeps on the futex if it reaches that futex before the end of the parent&#8217;s sleep (likely). If it did reach that <code>FUTEX_WAIT</code> there, it gets awoken by the parent.</p>
<div class="paragraph">
<p>So what you see is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>main start
child start
[wait 1s]
parent after sleep
child after parent sleep</pre>
</div>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="userland-mutex-implementation"><a class="anchor" href="#userland-mutex-implementation"></a><a class="link" href="#userland-mutex-implementation">28.7.1.1. Userland mutex implementation</a></h5>
<div class="paragraph">
<p>The best article to understand spinlocks is: <a href="https://eli.thegreenplace.net/2018/basics-of-futexes/" class="bare">https://eli.thegreenplace.net/2018/basics-of-futexes/</a></p>
</div>
<div class="paragraph">
<p>The example in <code>man futex</code> is also a must.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="getcpu"><a class="anchor" href="#getcpu"></a><a class="link" href="#getcpu">28.7.2. <code>getcpu</code> system call and the <code>sched_getaffinity</code> glibc wrapper</a></h4>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sched_getcpu.c">userland/linux/sched_getcpu.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/getcpu.c">userland/linux/getcpu.c</a>: a wrapper close the the syscall that also returns the current NUMA node</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/getcpu_syscall.c">userland/linux/getcpu_syscall.c</a>: the wrapper segfaults on error handling, so double checking with the real syscall: <a href="https://stackoverflow.com/questions/9260937/unix-socket-error-14-efault-bad-address/61879849#61879849" class="bare">https://stackoverflow.com/questions/9260937/unix-socket-error-14-efault-bad-address/61879849#61879849</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sched_getcpu_barrier.c">userland/linux/sched_getcpu_barrier.c</a>: this uses a barrier to ensure that gem5 will run each thread on one separate CPU</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Returns the CPU that the process/thread is currently running on:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/491520/how-can-i-get-the-cpu-core-number-from-within-a-user-space-app-linux-c" class="bare">https://stackoverflow.com/questions/491520/how-can-i-get-the-cpu-core-number-from-within-a-user-space-app-linux-c</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/6026896/how-to-know-on-which-physical-processor-and-on-which-physical-core-my-code-is-ru/16574301#16574301" class="bare">https://stackoverflow.com/questions/6026896/how-to-know-on-which-physical-processor-and-on-which-physical-core-my-code-is-ru/16574301#16574301</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>So when running a multicore program, we may see that each thread can be running on a different core.</p>
</div>
<div class="paragraph">
<p>The cores in which the process runs can be fixed with <code>sched_setaffinity</code> as shown at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/sched_getaffinity.c">userland/linux/sched_getaffinity.c</a>.</p>
</div>
<div class="paragraph">
<p>So when I run it with <code>main</code> thread + 4 threads on a 4 core CPUs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./userland/linux/sched_getcpu.out 4</pre>
</div>
</div>
<div class="paragraph">
<p>I see random outputs like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>7
2
1
5</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>5
0
2
1</pre>
</div>
</div>
<div class="paragraph">
<p>Due to the way that <a href="#gem5-syscall-emulation-multithreading">gem5 syscall emulation multithreading</a> however, the output is more deterministic in that case, see that section for further details.</p>
</div>
</div>
<div class="sect3">
<h4 id="perf-event-open"><a class="anchor" href="#perf-event-open"></a><a class="link" href="#perf-event-open">28.7.3. <code>perf_event_open</code> system call</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/perf_event_open.c">userland/linux/perf_event_open.c</a></p>
</div>
<div class="paragraph">
<p>On ARM, <code>perf_event_open</code> uses the <a href="#arm-pmu">ARM PMU</a>. The mapping between kernel events and ARM PMU events can be found at: <a href="https://github.com/cirosantilli/linux/blob/v5.9/arch/arm64/kernel/perf_event.c" class="bare">https://github.com/cirosantilli/linux/blob/v5.9/arch/arm64/kernel/perf_event.c</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>man perf_event_open</code></p>
</li>
<li>
<p><a href="https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/using-the-arm-performance-monitor-unit-pmu-linux-driver" class="bare">https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/using-the-arm-performance-monitor-unit-pmu-linux-driver</a></p>
</li>
<li>
<p>instruction counts: <a href="https://stackoverflow.com/questions/13313510/quick-way-to-count-number-of-instructions-executed-in-a-c-program/64863392#64863392" class="bare">https://stackoverflow.com/questions/13313510/quick-way-to-count-number-of-instructions-executed-in-a-c-program/64863392#64863392</a></p>
</li>
<li>
<p>cycle counts:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/13772567/how-to-get-the-cpu-cycle-count-in-x86-64-from-c/64898073#64898073" class="bare">https://stackoverflow.com/questions/13772567/how-to-get-the-cpu-cycle-count-in-x86-64-from-c/64898073#64898073</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/3830883/cpu-cycle-count-based-profiling-in-c-c-linux-x86-64/64898121#64898121" class="bare">https://stackoverflow.com/questions/3830883/cpu-cycle-count-based-profiling-in-c-c-linux-x86-64/64898121#64898121</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/35923834/what-is-the-most-reliable-way-to-measure-the-number-of-cycles-of-my-program-in-c/64898206#64898206" class="bare">https://stackoverflow.com/questions/35923834/what-is-the-most-reliable-way-to-measure-the-number-of-cycles-of-my-program-in-c/64898206#64898206</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/352166/measure-exact-clock-cycles-for-a-c-assembly-program/620317#620317" class="bare">https://unix.stackexchange.com/questions/352166/measure-exact-clock-cycles-for-a-c-assembly-program/620317#620317</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/8522140/linux-alternative-to-windows-high-resolution-performance-counter-api/64898303#64898303" class="bare">https://stackoverflow.com/questions/8522140/linux-alternative-to-windows-high-resolution-performance-counter-api/64898303#64898303</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>cache misses: <a href="https://stackoverflow.com/questions/10082517/simplest-tool-to-measure-c-program-cache-hit-miss-and-cpu-time-in-linux/64899613#64899613" class="bare">https://stackoverflow.com/questions/10082517/simplest-tool-to-measure-c-program-cache-hit-miss-and-cpu-time-in-linux/64899613#64899613</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="linux-calling-conventions"><a class="anchor" href="#linux-calling-conventions"></a><a class="link" href="#linux-calling-conventions">28.8. Linux calling conventions</a></h3>
<div class="paragraph">
<p>A summary of results is shown at: <a href="#table-linux-calling-conventions">Table 3, &#8220;Summary of Linux calling conventions for several architectures&#8221;</a>.</p>
</div>
<table id="table-linux-calling-conventions" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 3. Summary of Linux calling conventions for several architectures</caption>
<colgroup>
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
<col style="width: 25%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">arch</th>
<th class="tableblock halign-left valign-top">arguments</th>
<th class="tableblock halign-left valign-top">return value</th>
<th class="tableblock halign-left valign-top">callee saved registers</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">x86_64</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">rdi, rsi, rdx, rcx, r8, r9, xmm0–7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">rax, rdx</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">rbx, rbp, r12–r15</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">arm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">r0-r3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">r0-r3</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">r4-r11</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">aarch64</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">x0-x7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">x0-x7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">x19-x29</p></td>
</tr>
</tbody>
</table>
<div class="sect3">
<h4 id="x86-64-calling-convention"><a class="anchor" href="#x86-64-calling-convention"></a><a class="link" href="#x86-64-calling-convention">28.8.1. x86_64 calling convention</a></h4>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/x86_64.h">lkmc/x86_64.h</a> <code>ENTRY</code> and <code>EXIT</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>One important catch is that the stack must always be aligned to 16-bits before making calls: <a href="https://stackoverflow.com/questions/56324948/why-does-calling-the-c-abort-function-from-an-x86-64-assembly-function-lead-to" class="bare">https://stackoverflow.com/questions/56324948/why-does-calling-the-c-abort-function-from-an-x86-64-assembly-function-lead-to</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI" class="bare">https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/18024672/what-registers-are-preserved-through-a-linux-x86-64-function-call/55207335#55207335" class="bare">https://stackoverflow.com/questions/18024672/what-registers-are-preserved-through-a-linux-x86-64-function-call/55207335#55207335</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="arm-calling-convention"><a class="anchor" href="#arm-calling-convention"></a><a class="link" href="#arm-calling-convention">28.8.2. ARM calling convention</a></h4>
<div class="paragraph">
<p>Call C standard library functions from assembly and vice versa.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>arm</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/arm.h">lkmc/arm.h</a> <code>ENTRY</code> and <code>EXIT</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/linux/c_from_asm.S">userland/arch/arm/linux/c_from_asm.S</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>aarch64</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/aarch64.h">lkmc/aarch64.h</a> <code>ENTRY</code> and <code>EXIT</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/linux/asm_from_c.c">userland/arch/aarch64/inline_asm/linux/asm_from_c.c</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>ARM Architecture Procedure Call Standard (AAPCS) is the name that ARM Holdings gives to the calling convention.</p>
</div>
<div class="paragraph">
<p>Official specification: <a href="http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042f/IHI0042F_aapcs.pdf" class="bare">http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042f/IHI0042F_aapcs.pdf</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/Calling_convention#ARM_(A32" class="bare">https://en.wikipedia.org/wiki/Calling_convention#ARM_(A32</a>) Wiki contains the master list as usual.</p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/8422287/calling-c-functions-from-arm-assembly" class="bare">https://stackoverflow.com/questions/8422287/calling-c-functions-from-arm-assembly</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/261419/arm-to-c-calling-convention-registers-to-save" class="bare">https://stackoverflow.com/questions/261419/arm-to-c-calling-convention-registers-to-save</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/10494848/arm-whats-the-difference-between-apcs-and-aapcs-abi" class="bare">https://stackoverflow.com/questions/10494848/arm-whats-the-difference-between-apcs-and-aapcs-abi</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gnu-gas-assembler"><a class="anchor" href="#gnu-gas-assembler"></a><a class="link" href="#gnu-gas-assembler">28.9. GNU GAS assembler</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/GNU_Assembler">GNU GAS</a> is the default assembler used by GDB, and therefore it completely dominates in Linux.</p>
</div>
<div class="paragraph">
<p>The Linux kernel in particular uses GNU GAS assembly extensively for the arch specific parts under <code>arch/</code>.</p>
</div>
<div class="sect3">
<h4 id="gnu-gas-assembler-comments"><a class="anchor" href="#gnu-gas-assembler-comments"></a><a class="link" href="#gnu-gas-assembler-comments">28.9.1. GNU GAS assembler comments</a></h4>
<div class="paragraph">
<p>In this tutorial, we use exclusively C Preprocessor <code>/**/</code> comments because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>they are the same for all archs</p>
</li>
<li>
<p>we are already stuck to the C Preprocessor because GNU GAS macros are unusable so we need <code>#define</code></p>
</li>
<li>
<p>mixing <code>#</code> GNU GAS comments and <code>#define</code> is a bad idea ;-)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>But just in case you want to suffer, see this full explanation of GNU GAS comments: <a href="https://stackoverflow.com/questions/15663280/how-to-make-the-gnu-assembler-use-a-slash-for-comments/51991349#51991349" class="bare">https://stackoverflow.com/questions/15663280/how-to-make-the-gnu-assembler-use-a-slash-for-comments/51991349#51991349</a></p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/comments.S">userland/arch/arm/comments.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/comments.S">userland/arch/aarch64/comments.S</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gnu-gas-assembler-immediates"><a class="anchor" href="#gnu-gas-assembler-immediates"></a><a class="link" href="#gnu-gas-assembler-immediates">28.9.2. GNU GAS assembler immediates</a></h4>
<div class="paragraph">
<p>Summary:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>x86 always dollar <code>$</code> everywhere.</p>
</li>
<li>
<p>ARM: can use either <code>#</code>, <code>$</code> or nothing depending on v7 vs v8 and <a href="#gnu-gas-assembler-arm-unified-syntax"><code>.syntax unified</code></a>.</p>
<div class="paragraph">
<p>Fuller explanation at: <a href="https://stackoverflow.com/questions/21652884/is-the-hash-required-for-immediate-values-in-arm-assembly/51987780#51987780" class="bare">https://stackoverflow.com/questions/21652884/is-the-hash-required-for-immediate-values-in-arm-assembly/51987780#51987780</a></p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/immediates.S">userland/arch/arm/immediates.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/immediates.S">userland/arch/aarch64/immediates.S</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="gnu-gas-assembler-data-sizes"><a class="anchor" href="#gnu-gas-assembler-data-sizes"></a><a class="link" href="#gnu-gas-assembler-data-sizes">28.9.3. GNU GAS assembler data sizes</a></h4>
<div class="paragraph">
<p>Let&#8217;s see how many bytes go into each data type:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/gas_data_sizes.S">userland/arch/x86_64/gas_data_sizes.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/gas_data_sizes.S">userland/arch/arm/gas_data_sizes.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/gas_data_sizes.S">userland/arch/aarch64/gas_data_sizes.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The results are shown at: <a href="#table-gas-data-sizes">Table 4, &#8220;Summary of GNU GAS assembler data sizes&#8221;</a>.</p>
</div>
<table id="table-gas-data-sizes" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 4. Summary of GNU GAS assembler data sizes</caption>
<colgroup>
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
<col style="width: 20%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">.byte</th>
<th class="tableblock halign-left valign-top">.word</th>
<th class="tableblock halign-left valign-top">.long</th>
<th class="tableblock halign-left valign-top">.quad</th>
<th class="tableblock halign-left valign-top">.octa</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">x86</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">8</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">16</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">arm</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">8</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">16</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">aarch64</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">4</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>and also keep in mind that according to the manual:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>.int</code> is the same as <code>.long</code></p>
</li>
<li>
<p><code>.hword</code> is the same as <code>.short</code> which is usually the same as <code>.word</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://sourceware.org/binutils/docs-2.32/as/Pseudo-Ops.html#Pseudo-Ops" class="bare">https://sourceware.org/binutils/docs-2.32/as/Pseudo-Ops.html#Pseudo-Ops</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/43005411/how-does-the-quad-directive-work-in-assembly/43006616" class="bare">https://stackoverflow.com/questions/43005411/how-does-the-quad-directive-work-in-assembly/43006616</a></p>
</li>
<li>
<p><a href="https://gist.github.com/steakknife/d47d0b19a24817f48027" class="bare">https://gist.github.com/steakknife/d47d0b19a24817f48027</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="gnu-gas-assembler-arm-specifics"><a class="anchor" href="#gnu-gas-assembler-arm-specifics"></a><a class="link" href="#gnu-gas-assembler-arm-specifics">28.9.3.1. GNU GAS assembler ARM specifics</a></h5>
<div class="sect5">
<h6 id="gnu-gas-assembler-arm-unified-syntax"><a class="anchor" href="#gnu-gas-assembler-arm-unified-syntax"></a><a class="link" href="#gnu-gas-assembler-arm-unified-syntax">28.9.3.1.1. GNU GAS assembler ARM unified syntax</a></h6>
<div class="paragraph">
<p>There are two types of ARMv7 assemblies:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>.syntax divided</code></p>
</li>
<li>
<p><code>.syntax unified</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>They are very similar, but unified is the new and better one, which we use in this tutorial.</p>
</div>
<div class="paragraph">
<p>Unfortunately, for backwards compatibility, GNU AS 2.31.1 and GCC 8.2.0 still use <code>.syntax divided</code> by default.</p>
</div>
<div class="paragraph">
<p>The concept of unified assembly is mentioned in ARM&#8217;s official assembler documentation: <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0473c/BABJIHGJ.html" class="bare">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0473c/BABJIHGJ.html</a> and is often called Unified Assembly Language (UAL).</p>
</div>
<div class="paragraph">
<p>Some of the differences include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>#</code> is optional in unified syntax int literals, see <a href="#gnu-gas-assembler-immediates">GNU GAS assembler immediates</a></p>
</li>
<li>
<p>many mnemonics changed:</p>
<div class="ulist">
<ul>
<li>
<p>most of them are condition code position changes, e.g. ANDSEQ vs ANDEQS: <a href="https://stackoverflow.com/questions/51184921/wierd-gcc-behaviour-with-arm-assembler-andseq-instruction" class="bare">https://stackoverflow.com/questions/51184921/wierd-gcc-behaviour-with-arm-assembler-andseq-instruction</a></p>
</li>
<li>
<p>but there are some more drastic ones, e.g. SWI vs <a href="#arm-svc-instruction">SVC</a>: <a href="https://stackoverflow.com/questions/8459279/are-arm-instructuons-swi-and-svc-exactly-same-thing/54078731#54078731" class="bare">https://stackoverflow.com/questions/8459279/are-arm-instructuons-swi-and-svc-exactly-same-thing/54078731#54078731</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>cannot have implicit destination with shift, see: <a href="#arm-shift-suffixes">Section 30.4.4.1, &#8220;ARM shift suffixes&#8221;</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect4">
<h5 id="gnu-gas-assembler-arm-n-and-w-suffixes"><a class="anchor" href="#gnu-gas-assembler-arm-n-and-w-suffixes"></a><a class="link" href="#gnu-gas-assembler-arm-n-and-w-suffixes">28.9.3.2. GNU GAS assembler ARM .n and .w suffixes</a></h5>
<div class="paragraph">
<p>When reading disassembly, many instructions have either a <code>.n</code> or <code>.w</code> suffix.</p>
</div>
<div class="paragraph">
<p><code>.n</code> means narrow, and stands for the <a href="#arm-instruction-encodings">Thumb encoding</a> of an instructions, while <code>.w</code> means wide and stands for the ARM encoding.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/27147043/n-suffix-to-branch-instruction" class="bare">https://stackoverflow.com/questions/27147043/n-suffix-to-branch-instruction</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gnu-gas-assembler-char-literals"><a class="anchor" href="#gnu-gas-assembler-char-literals"></a><a class="link" href="#gnu-gas-assembler-char-literals">28.9.4. GNU GAS assembler char literals</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/char_literals.S">userland/arch/x86_64/char_literals.S</a></p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/33246811/how-to-use-character-literals-in-gnu-gas-to-replace-numbers" class="bare">https://stackoverflow.com/questions/33246811/how-to-use-character-literals-in-gnu-gas-to-replace-numbers</a></p>
</div>
<div class="paragraph">
<p>This syntax plays horribly with the C preprocessor:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>MACRO($'a)</pre>
</div>
</div>
<div class="paragraph">
<p>fails because cpp treats string and char literals magically.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="nop-instructions"><a class="anchor" href="#nop-instructions"></a><a class="link" href="#nop-instructions">28.10. NOP instructions</a></h3>
<div class="ulist">
<ul>
<li>
<p>x86: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/nop.S">NOP</a></p>
</li>
<li>
<p>ARM: <a href="#arm-nop-instruction">Section 30.5.1, &#8220;ARM NOP instruction&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>No OPeration.</p>
</div>
<div class="paragraph">
<p>Does nothing except take up one processor cycle and occupy some instruction memory.</p>
</div>
<div class="paragraph">
<p>Applications: <a href="https://stackoverflow.com/questions/234906/whats-the-purpose-of-the-nop-opcode" class="bare">https://stackoverflow.com/questions/234906/whats-the-purpose-of-the-nop-opcode</a></p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="x86-userland-assembly"><a class="anchor" href="#x86-userland-assembly"></a><a class="link" href="#x86-userland-assembly">29. x86 userland assembly</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Arch agnostic infrastructure getting started at: <a href="#userland-assembly">Section 28, &#8220;Userland assembly&#8221;</a>.</p>
</div>
<div class="sect2">
<h3 id="x86-registers"><a class="anchor" href="#x86-registers"></a><a class="link" href="#x86-registers">29.1. x86 registers</a></h3>
<div class="paragraph">
<p>link:userland/arch/x86_64/registers.S</p>
</div>
<div class="literalblock">
<div class="content">
<pre>|-----------------------------------------------|
|  7  |  6  |  5  |  4  |  3  |  2  |  1  |  0  |
|-----------------------------------------------|
|                       |           | AH  | AL  |
|-----------------------------------------------|
|                       |           |    AX     |
|-----------------------------------------------|
|                       |          EAX          |
|-----------------------------------------------|
|                      RAX                      |
|-----------------------------------------------|</pre>
</div>
</div>
<div class="paragraph">
<p>For the newer x86_64 registers, the naming convention is somewhat saner:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>|-----------------------------------------------|
|  7  |  6  |  5  |  4  |  3  |  2  |  1  |  0  |
|-----------------------------------------------|
|                       |           |R12H |R12L |
|-----------------------------------------------|
|                       |           |    R12W   |
|-----------------------------------------------|
|                       |          R12D         |
|-----------------------------------------------|
|                      R12                      |
|-----------------------------------------------|</pre>
</div>
</div>
<div class="paragraph">
<p>Most of the 8 older x86 general purpose registers are not "really" general purpose in the sense that a few instructions magically use them without an explicit encoding. This is reflected in their names:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>RAX: Accumulator. The general place where you add, subtract and otherwise manipulate results in-place. Magic for example for <a href="#x86-binary-arithmetic-instructions">MUL</a>.</p>
</li>
<li>
<p>RCX, RSI, RDI: Counter, Source and Destination. Used in <a href="#x86-string-instructions">x86 string instructions</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="x86-flags-registers"><a class="anchor" href="#x86-flags-registers"></a><a class="link" href="#x86-flags-registers">29.1.1. x86 FLAGS registers</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/FLAGS_register" class="bare">https://en.wikipedia.org/wiki/FLAGS_register</a></p>
</div>
<div class="paragraph">
<p>TODO: add some more info here. Just need a link placeholder for now.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-addressing-modes"><a class="anchor" href="#x86-addressing-modes"></a><a class="link" href="#x86-addressing-modes">29.2. x86 addressing modes</a></h3>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/address_modes.S">userland/arch/x86_64/address_modes.S</a></p>
</div>
<div class="paragraph">
<p>Several x86 instructions can calculate addresses of a complex form:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>s:a(b, c, d)</pre>
</div>
</div>
<div class="paragraph">
<p>which expands to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>a + b + c * d</pre>
</div>
</div>
<div class="paragraph">
<p>Where the instruction encoding allows for:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>a</code>: any 8 or 32-bit general purpose register</p>
</li>
<li>
<p><code>b</code>: any 32-bit general purpose register except ESP</p>
</li>
<li>
<p><code>c</code>: 1, 2, 4 or 8 (encoded in 2 SIB bits)</p>
</li>
<li>
<p><code>d</code>: immediate constant</p>
</li>
<li>
<p><code>s</code>: a segment register. Cannot be tested simply from userland, so we won&#8217;t talk about them here. See: <a href="https://github.com/cirosantilli/x86-bare-metal-examples/blob/6606a2647d44bc14e6fd695c0ea2b6b7a5f04ca3/segment_registers_real_mode.S" class="bare">https://github.com/cirosantilli/x86-bare-metal-examples/blob/6606a2647d44bc14e6fd695c0ea2b6b7a5f04ca3/segment_registers_real_mode.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The common compiler usage is:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>a</code>: base pointer</p>
</li>
<li>
<p><code>b</code>: array offset</p>
</li>
<li>
<p><code>c</code> and <code>d</code>: struct offset</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 3.7.5 "Specifying an Offset"</p>
</li>
<li>
<p><a href="https://sourceware.org/binutils/docs-2.18/as/i386_002dMemory.html" class="bare">https://sourceware.org/binutils/docs-2.18/as/i386_002dMemory.html</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="x86-data-transfer-instructions"><a class="anchor" href="#x86-data-transfer-instructions"></a><a class="link" href="#x86-data-transfer-instructions">29.3. x86 data transfer instructions</a></h3>
<div class="paragraph">
<p>5.1.1 "Data Transfer Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/lea.S">userland/arch/x86_64/lea.S</a>: LEA</p>
</li>
<li>
<p>Integer typecasts</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movzx.S">userland/arch/x86_64/movzx.S</a>: MOVZX</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movsx.S">userland/arch/x86_64/movsx.S</a>: MOVSX</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/bswap.S">userland/arch/x86_64/bswap.S</a>: BSWAP: convert between little endian and big endian</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/pushf.S">userland/arch/x86_64/pushf.S</a> PUSHF: <a href="#x86-push-and-pop-instructions">push and pop</a> the <a href="#x86-flags-registers">x86 FLAGS registers</a> to / from the stack</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="x86-exchange-instructions"><a class="anchor" href="#x86-exchange-instructions"></a><a class="link" href="#x86-exchange-instructions">29.3.1. x86 exchange instructions</a></h4>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 7.3.1.2 "Exchange Instructions":</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/xadd.S">userland/arch/x86_64/xadd.S</a> XADD: exchange and add. This is how C `&lt;atomic&gt;`'s' `` is implemented in GCC 5.1. TODO: why is the exchange part needed?</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/xchg.S">userland/arch/x86_64/xchg.S</a> XCHG: exchange two values</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO: concrete multi-thread <a href="#gcc-inline-assembly">GCC inline assembly</a> examples of how all those instructions are normally used as synchronization primitives.</p>
</div>
<div class="sect4">
<h5 id="x86-cmpxchg-instruction"><a class="anchor" href="#x86-cmpxchg-instruction"></a><a class="link" href="#x86-cmpxchg-instruction">29.3.1.1. x86 CMPXCHG instruction</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cmpxchg.S">userland/arch/x86_64/cmpxchg.S</a></p>
</div>
<div class="paragraph">
<p>CMPXCHG: compare and exchange. <code>cmpxchg a, b</code> does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if (RAX == b) {
    ZF = 1
    b = a
} else {
    ZF = 0
    RAX = b
}</pre>
</div>
</div>
<div class="paragraph">
<p>TODO application: <a href="https://stackoverflow.com/questions/6935442/x86-spinlock-using-cmpxchg" class="bare">https://stackoverflow.com/questions/6935442/x86-spinlock-using-cmpxchg</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="x86-push-and-pop-instructions"><a class="anchor" href="#x86-push-and-pop-instructions"></a><a class="link" href="#x86-push-and-pop-instructions">29.3.2. x86 PUSH and POP instructions</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/push.S">userland/arch/x86_64/push.S</a></p>
</div>
<div class="paragraph">
<p><code>push %rax</code> is basically equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sub $8, %rsp
mov %rax, (%rsp)</pre>
</div>
</div>
<div class="paragraph">
<p>and <code>pop %rax</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov (%rsp), %rax
add $8, %rsp</pre>
</div>
</div>
<div class="paragraph">
<p>Why do those instructions exist at all vs MOV / ADD / SUB: <a href="https://stackoverflow.com/questions/4584089/what-is-the-function-of-push-pop-registers-in-x86-assembly/33583134#33583134" class="bare">https://stackoverflow.com/questions/4584089/what-is-the-function-of-push-pop-registers-in-x86-assembly/33583134#33583134</a></p>
</div>
</div>
<div class="sect3">
<h4 id="x86-cqto-and-cltq-instructions"><a class="anchor" href="#x86-cqto-and-cltq-instructions"></a><a class="link" href="#x86-cqto-and-cltq-instructions">29.3.3. x86 CQTO and CLTQ instructions</a></h4>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cqto.S">userland/arch/x86_64/cqto.S</a> CQTO</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cltq.S">userland/arch/x86_64/cltq.S</a> CLTQ</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Instructions without E suffix: sign extend RAX into RDX:RAX.</p>
</div>
<div class="paragraph">
<p>Instructions E suffix: sign extend withing RAX itself.</p>
</div>
<div class="paragraph">
<p>Common combo with IDIV 32-bit, which takes the input from EDX:EAX: so you need to set up EDX before calling it.</p>
</div>
<div class="paragraph">
<p>Has some Intel vs AT&amp;T name overload hell:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/6555094/what-does-cltq-do-in-assembly/45386217#45386217" class="bare">https://stackoverflow.com/questions/6555094/what-does-cltq-do-in-assembly/45386217#45386217</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/17170388/trying-to-understand-the-assembly-instruction-cltd-on-x86/50315201#50315201" class="bare">https://stackoverflow.com/questions/17170388/trying-to-understand-the-assembly-instruction-cltd-on-x86/50315201#50315201</a></p>
</li>
<li>
<p><a href="https://sourceware.org/binutils/docs/as/i386_002dMnemonics.html" class="bare">https://sourceware.org/binutils/docs/as/i386_002dMnemonics.html</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>GNU GAS accepts both syntaxes, see: <a href="#table-cqto-cltq">Table 5, &#8220;CQTO and CLTQ family Intel vs AT&amp;T&#8221;</a>.</p>
</div>
<table id="table-cqto-cltq" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 5. CQTO and CLTQ family Intel vs AT&amp;T</caption>
<colgroup>
<col style="width: 33.3333%;">
<col style="width: 33.3333%;">
<col style="width: 33.3334%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Intel</th>
<th class="tableblock halign-left valign-top">AT&amp;T</th>
<th class="tableblock halign-left valign-top">From</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">To</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CBW</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CBTW</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">AL</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">AX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CWDE</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CWTL</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">AX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">EAX</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CWD</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CWTD</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">AX</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">DX:AX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CDQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CLTD</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">EAX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">EDX:EAX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CDQE</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CLTQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">EAX</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">RAX</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">CQO</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">CQTO</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">RAX</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect3">
<h4 id="x86-cmovcc-instructions"><a class="anchor" href="#x86-cmovcc-instructions"></a><a class="link" href="#x86-cmovcc-instructions">29.3.4. x86 CMOVcc instructions</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cmovcc.S">userland/arch/x86_64/cmovcc.S</a>: CMOVcc</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>mov if a condition is met:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CMOVcc a, b</pre>
</div>
</div>
<div class="paragraph">
<p>Equals:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>if(flag) a = b</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>cc</code> are the same flags as Jcc.</p>
</div>
<div class="paragraph">
<p>Vs jmp:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/14131096/why-is-a-conditional-move-not-vulnerable-for-branch-prediction-failure" class="bare">https://stackoverflow.com/questions/14131096/why-is-a-conditional-move-not-vulnerable-for-branch-prediction-failure</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/27136961/what-is-it-about-cmov-which-improves-cpu-pipeline-performance" class="bare">https://stackoverflow.com/questions/27136961/what-is-it-about-cmov-which-improves-cpu-pipeline-performance</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/26154488/difference-between-conditional-instructions-cmov-and-jump-instructions" class="bare">https://stackoverflow.com/questions/26154488/difference-between-conditional-instructions-cmov-and-jump-instructions</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/6754454/speed-difference-between-if-else-and-ternary-operator-in-c?lq=1#comment8007791_6754495" class="bare">https://stackoverflow.com/questions/6754454/speed-difference-between-if-else-and-ternary-operator-in-c?lq=1#comment8007791_6754495</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Not necessarily faster because of branch prediction.</p>
</div>
<div class="paragraph">
<p>This is partly why the ternary <code>?</code> C operator exists: <a href="https://stackoverflow.com/questions/3565368/ternary-operator-vs-if-else" class="bare">https://stackoverflow.com/questions/3565368/ternary-operator-vs-if-else</a></p>
</div>
<div class="paragraph">
<p>It is interesting to compare this with ARMv7 conditional execution: which is available for all instructions, as shown at: <a href="#arm-conditional-execution">Section 30.2.5, &#8220;ARM conditional execution&#8221;</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-binary-arithmetic-instructions"><a class="anchor" href="#x86-binary-arithmetic-instructions"></a><a class="link" href="#x86-binary-arithmetic-instructions">29.4. x86 binary arithmetic instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.2 "Binary Arithmetic Instructions":</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/add.S">userland/arch/x86_64/add.S</a>: ADD</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/inc.S">userland/arch/x86_64/inc.S</a>: INC</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/adc.S">userland/arch/x86_64/adc.S</a>: ADC</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/sub.S">userland/arch/x86_64/sub.S</a>: SUB</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/dec.S">userland/arch/x86_64/dec.S</a>: DEC</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/sbb.S">userland/arch/x86_64/sbb.S</a>: SBB</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/mul.S">userland/arch/x86_64/mul.S</a>: MUL</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/neg.S">userland/arch/x86_64/neg.S</a>: NEG</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/imul.S">userland/arch/x86_64/imul.S</a>: IMUL</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/div.S">userland/arch/x86_64/div.S</a>: DIV</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/div_overflow.S">userland/arch/x86_64/div_overflow.S</a>: DIV overflow</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/div_zero.S">userland/arch/x86_64/div_zero.S</a>: DIV zero</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/idiv.S">userland/arch/x86_64/idiv.S</a>: IDIV</p>
</li>
</ul>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cmp.S">userland/arch/x86_64/cmp.S</a>: CMP</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="x86-logical-instructions"><a class="anchor" href="#x86-logical-instructions"></a><a class="link" href="#x86-logical-instructions">29.5. x86 logical instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.4 "Logical Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/and.S">userland/arch/x86_64/and.S</a>: AND</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/not.S">userland/arch/x86_64/not.S</a>: NOT</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/or.S">userland/arch/x86_64/or.S</a>: OR</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/xor.S">userland/arch/x86_64/xor.S</a>: XOR</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="x86-shift-and-rotate-instructions"><a class="anchor" href="#x86-shift-and-rotate-instructions"></a><a class="link" href="#x86-shift-and-rotate-instructions">29.6. x86 shift and rotate instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.5 "Shift and Rotate Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/shl.S">SHL and SHR</a></p>
<div class="paragraph">
<p>SHift left or Right and insert 0.</p>
</div>
<div class="paragraph">
<p>CF == the bit that got shifted out.</p>
</div>
<div class="paragraph">
<p>Application: quick unsigned multiply and divide by powers of 2.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/sal.S">SAL and SAR</a></p>
<div class="paragraph">
<p>Application: signed multiply and divide by powers of 2.</p>
</div>
<div class="paragraph">
<p>Mnemonics: Shift Arithmetic Left and Right</p>
</div>
<div class="paragraph">
<p>Keeps the same sign on right shift.</p>
</div>
<div class="paragraph">
<p>Not directly exposed in C, for which signed shift is undetermined behavior, but does exist in Java via the <code>&gt;&gt;&gt;</code> operator. C compilers can omit it however.</p>
</div>
<div class="paragraph">
<p>SHL and SAL are exactly the same and have the same encoding: <a href="https://stackoverflow.com/questions/8373415/difference-between-shl-and-sal-in-80x86/56621271#56621271" class="bare">https://stackoverflow.com/questions/8373415/difference-between-shl-and-sal-in-80x86/56621271#56621271</a></p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rol.S">userland/arch/x86_64/rol.S</a>: ROL and ROR</p>
<div class="paragraph">
<p>Rotates the bit that is going out around to the other side.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rol.S">userland/arch/x86_64/rol.S</a>: RCL and RCR</p>
<div class="paragraph">
<p>Like ROL and ROR, but insert the carry bit instead, which effectively generates a rotation of 8 + 1 bits. TODO application.</p>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="x86-bit-and-byte-instructions"><a class="anchor" href="#x86-bit-and-byte-instructions"></a><a class="link" href="#x86-bit-and-byte-instructions">29.7. x86 bit and byte instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.6 "Bit and Byte Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/bt.S">userland/arch/x86_64/bt.S</a>: BT</p>
<div class="paragraph">
<p>Bit test: test if the Nth bit a bit of a register is set and store the result in the CF FLAG.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CF = reg[N]</pre>
</div>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/btr.S">userland/arch/x86_64/btr.S</a>: BTR</p>
<div class="paragraph">
<p>Do a BT and then set the bit to 0.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/btc.S">userland/arch/x86_64/btc.S</a>: BTC</p>
<div class="paragraph">
<p>Do a BT and then swap the value of the tested bit.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/setcc.S">userland/arch/x86_64/setcc.S</a>: SETcc</p>
<div class="paragraph">
<p>Set a byte of a register to 0 or 1 depending on the cc condition.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/1406783/how-to-read-and-write-x86-flags-registers-directly/30952577#30952577" class="bare">https://stackoverflow.com/questions/1406783/how-to-read-and-write-x86-flags-registers-directly/30952577#30952577</a></p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/popcnt.S">userland/arch/x86_64/popcnt.S</a>: POPCNT</p>
<div class="paragraph">
<p>Count the number of 1 bits.</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/test.S">userland/arch/x86_64/test.S</a>: TEST</p>
<div class="paragraph">
<p>Like <a href="#x86-binary-arithmetic-instructions">CMP</a> but does AND instead of SUB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ZF = (!(X &amp;&amp; Y)) ? 1 : 0</pre>
</div>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="x86-control-transfer-instructions"><a class="anchor" href="#x86-control-transfer-instructions"></a><a class="link" href="#x86-control-transfer-instructions">29.8. x86 control transfer instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.7 "Control Transfer Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/jmp.S">userland/arch/x86_64/jmp.S</a>: JMP</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/jmp_indirect.S">userland/arch/x86_64/jmp_indirect.S</a>: JMP indirect</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="x86-jcc-instructions"><a class="anchor" href="#x86-jcc-instructions"></a><a class="link" href="#x86-jcc-instructions">29.8.1. x86 Jcc instructions</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/jcc.S">userland/arch/x86_64/jcc.S</a></p>
</div>
<div class="paragraph">
<p>Jump if certain conditions of the flags register are met.</p>
</div>
<div class="paragraph">
<p>Jcc includes the instructions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>JZ, JNZ</p>
<div class="ulist">
<ul>
<li>
<p>JE, JNE: same as JZ, with two separate manual entries that say almost the same thing, lol: <a href="https://stackoverflow.com/questions/14267081/difference-between-je-jne-and-jz-jnz/14267662#14267662" class="bare">https://stackoverflow.com/questions/14267081/difference-between-je-jne-and-jz-jnz/14267662#14267662</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>JG: greater than, signed</p>
<div class="ulist">
<ul>
<li>
<p>JA: Above: greater than, unsigned</p>
</li>
</ul>
</div>
</li>
<li>
<p>JL: less than, signed</p>
<div class="ulist">
<ul>
<li>
<p>JB below: less than, unsigned</p>
</li>
</ul>
</div>
</li>
<li>
<p>JC: carry</p>
</li>
<li>
<p>JO: overflow</p>
</li>
<li>
<p>JP: parity. Why it exists: <a href="https://stackoverflow.com/questions/25707130/what-is-the-purpose-of-the-parity-flag-on-a-cpu" class="bare">https://stackoverflow.com/questions/25707130/what-is-the-purpose-of-the-parity-flag-on-a-cpu</a></p>
</li>
<li>
<p>JPE: parity even</p>
</li>
<li>
<p>JPO: parity odd</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>JG vs JA and JL vs JB:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/9617877/assembly-jg-jnle-jl-jnge-after-cmp/56613928#56613928" class="bare">https://stackoverflow.com/questions/9617877/assembly-jg-jnle-jl-jnge-after-cmp/56613928#56613928</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/20906639/difference-between-ja-and-jg-in-assembly" class="bare">https://stackoverflow.com/questions/20906639/difference-between-ja-and-jg-in-assembly</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="x86-loop-instruction"><a class="anchor" href="#x86-loop-instruction"></a><a class="link" href="#x86-loop-instruction">29.8.2. x86 LOOP instruction</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/loop.S">userland/arch/x86_64/loop.S</a></p>
</div>
<div class="paragraph">
<p>Vs <a href="#x86-jcc-instructions">Jcc</a>: <a href="https://stackoverflow.com/questions/6805692/x86-assembly-programming-loops-with-ecx-and-loop-instruction-versus-jmp-jcond" class="bare">https://stackoverflow.com/questions/6805692/x86-assembly-programming-loops-with-ecx-and-loop-instruction-versus-jmp-jcond</a> Holy CISC!</p>
</div>
</div>
<div class="sect3">
<h4 id="x86-string-instructions"><a class="anchor" href="#x86-string-instructions"></a><a class="link" href="#x86-string-instructions">29.8.3. x86 string instructions</a></h4>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.8 "String Instructions"</p>
</div>
<div class="paragraph">
<p>These instructions do some operation on an array item, and automatically update the index to the next item:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>First example explained in more detail</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/stos.S">userland/arch/x86_64/stos.S</a>: STOS: STOre String: store register to memory. STOSD is called STOSL in GNU GAS as usual: <a href="https://stackoverflow.com/questions/6211629/gcc-inline-assembly-error-no-such-instruction-stosd" class="bare">https://stackoverflow.com/questions/6211629/gcc-inline-assembly-error-no-such-instruction-stosd</a></p>
</li>
</ul>
</div>
</li>
<li>
<p>Further examples</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cmps.S">userland/arch/x86_64/cmps.S</a>: CMPS: CoMPare Strings: compare two values in memory with addresses given by RSI and RDI. Could be used to implement <code>memcmp</code>. Store the result in JZ as usual.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/lods.S">userland/arch/x86_64/lods.S</a>: LODS: LOaD String: load from memory to register.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movs.S">userland/arch/x86_64/movs.S</a>: MOVS: MOV String: move from one memory to another with addresses given by RSI and RDI. Could be used to implement <code>memmov</code>.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/scas.S">userland/arch/x86_64/scas.S</a>: SCAS: SCan String: compare memory to the value in a register. Could be used to implement <code>strchr</code>.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The RSI and RDI registers are actually named after these intructions! S is the source of string instructions, D is the destination of string instructions: <a href="https://stackoverflow.com/questions/1856320/purpose-of-esi-edi-registers" class="bare">https://stackoverflow.com/questions/1856320/purpose-of-esi-edi-registers</a></p>
</div>
<div class="paragraph">
<p>The direction of the index increment depends on the direction flag of the FLAGS register: 0 means forward and 1 means backward: <a href="https://stackoverflow.com/questions/9636691/what-are-cld-and-std-for-in-x86-assembly-language-what-does-df-do" class="bare">https://stackoverflow.com/questions/9636691/what-are-cld-and-std-for-in-x86-assembly-language-what-does-df-do</a></p>
</div>
<div class="paragraph">
<p>These instructions were originally developed to speed up "string" operations such as those present in the <code>&lt;string.h&gt;</code> header of the C standard library.</p>
</div>
<div class="paragraph">
<p>However, as computer architecture evolved, those instructions might not offer considerable speedups anymore, and modern glibc such as 2.29 just uses <a href="#x86-simd">x86 SIMD</a> operations instead:, see also: <a href="https://stackoverflow.com/questions/33480999/how-can-the-rep-stosb-instruction-execute-faster-than-the-equivalent-loop" class="bare">https://stackoverflow.com/questions/33480999/how-can-the-rep-stosb-instruction-execute-faster-than-the-equivalent-loop</a></p>
</div>
<div class="sect4">
<h5 id="x86-rep-prefix"><a class="anchor" href="#x86-rep-prefix"></a><a class="link" href="#x86-rep-prefix">29.8.3.1. x86 REP prefix</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rep.S">userland/arch/x86_64/rep.S</a></p>
</div>
<div class="paragraph">
<p>Repeat a string instruction RCX times:</p>
</div>
<div class="paragraph">
<p>As the repetitions happen:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>RCX decreases, until it reaches 0</p>
</li>
<li>
<p>RDI and RSI increase</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The variants: REPZ, REPNZ (alias REPE, REPNE) repeat a given instruction until something happens.</p>
</div>
<div class="paragraph">
<p>REP and REPZ also additionally stop if the comparison operation they repeat fails.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>REP: INS, OUTS, MOVS, LODS, and STOS</p>
</li>
<li>
<p>REPZ: CMPS and SCAS</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="x86-enter-and-leave-instructions"><a class="anchor" href="#x86-enter-and-leave-instructions"></a><a class="link" href="#x86-enter-and-leave-instructions">29.8.4. x86 ENTER and LEAVE instructions</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/enter.S">userland/arch/x86_64/enter.S</a></p>
</div>
<div class="paragraph">
<p>These instructions were designed to allocate and deallocate function stack frames in the prologue and epilogue: <a href="https://stackoverflow.com/questions/5959890/enter-vs-push-ebp-mov-ebp-esp-sub-esp-imm-and-leave-vs-mov-esp-ebp" class="bare">https://stackoverflow.com/questions/5959890/enter-vs-push-ebp-mov-ebp-esp-sub-esp-imm-and-leave-vs-mov-esp-ebp</a></p>
</div>
<div class="paragraph">
<p>ENTER appears obsolete and is kept mostly for backwards compatibility. LEAVE is still emitted by some compilers.</p>
</div>
<div class="paragraph">
<p>ENTER A, B is basically equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>push %rbp
mov %rsp, %rbp
sub %rsp, A</pre>
</div>
</div>
<div class="paragraph">
<p>which implies an allocation of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>one dword to remember EBP</p>
</li>
<li>
<p>A bytes for local function variables</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>I didn&#8217;t have the patience to study the B parameter, and it does not seem to be used often: <a href="https://stackoverflow.com/questions/26323215/do-any-languages-compilers-utilize-the-x86-enter-instruction-with-a-nonzero-ne" class="bare">https://stackoverflow.com/questions/26323215/do-any-languages-compilers-utilize-the-x86-enter-instruction-with-a-nonzero-ne</a></p>
</div>
<div class="paragraph">
<p>LEAVE is equivalent to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov %rbp, %rsp
pop %rbp</pre>
</div>
</div>
<div class="paragraph">
<p>which restores RSP and RBP to the values they had before the prologue.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-miscellaneous-instructions"><a class="anchor" href="#x86-miscellaneous-instructions"></a><a class="link" href="#x86-miscellaneous-instructions">29.9. x86 miscellaneous instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.13 "Miscellaneous Instructions"</p>
</div>
<div class="paragraph">
<p>NOP: <a href="#nop-instructions">Section 28.10, &#8220;NOP instructions&#8221;</a></p>
</div>
</div>
<div class="sect2">
<h3 id="x86-random-number-generator-instructions"><a class="anchor" href="#x86-random-number-generator-instructions"></a><a class="link" href="#x86-random-number-generator-instructions">29.10. x86 random number generator instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.1.15 Random Number Generator Instructions</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rdrand.S">userland/arch/x86_64/rdrand.S</a>: RDRAND</p>
</div>
<div class="paragraph">
<p>If you run that executable multiple times, it prints a random number every time to stdout.</p>
</div>
<div class="paragraph">
<p>RDRAND is a true random number generator!</p>
</div>
<div class="paragraph">
<p>This Intel engineer says its based on quantum effects: <a href="https://stackoverflow.com/questions/17616960/true-random-numbers-with-c11-and-rdrand/18004959#18004959" class="bare">https://stackoverflow.com/questions/17616960/true-random-numbers-with-c11-and-rdrand/18004959#18004959</a></p>
</div>
<div class="paragraph">
<p>Generated some polemic when kernel devs wanted to use it as part of <code>/dev/random</code>, because it could be used as a cryptographic backdoor by Intel since it is a black box.</p>
</div>
<div class="paragraph">
<p>RDRAND sets the carry flag when data is ready so we must loop if the carry flag isn&#8217;t set.</p>
</div>
<div class="sect3">
<h4 id="x86-cpuid-instruction"><a class="anchor" href="#x86-cpuid-instruction"></a><a class="link" href="#x86-cpuid-instruction">29.10.1. x86 CPUID instruction</a></h4>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cpuid.S">userland/arch/x86_64/cpuid.S</a></p>
</div>
<div class="paragraph">
<p>Fills EAX, EBX, ECX and EDX with CPU information.</p>
</div>
<div class="paragraph">
<p>The exact data to show depends on the value of EAX, and for a few cases instructions ECX. When it depends on ECX, it is called a sub-leaf. Out test program prints <code>eax == 0</code>.</p>
</div>
<div class="paragraph">
<p>On <a href="#p51">2017 Lenovo ThinkPad P51</a> for example the output EAX, EBX, ECX and EDX are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x00000016
0x756E6547
0x6C65746E
0x49656E69</pre>
</div>
</div>
<div class="paragraph">
<p>EBX and ECX are easy to interpret:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>EBX: 75 6e 65 47 == 'u', 'n', 'e', 'G' in ASCII</p>
</li>
<li>
<p>ECX: 6C 65 74 6E == 'l', 'e', 't', 'n'</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so we see the string <code>Genu ntel</code> which is a shorthand for "Genuine Intel". Ha, I wonder if they had serious CPU pirating problems in the past? :-)</p>
</div>
<div class="paragraph">
<p>Information available includes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>vendor</p>
</li>
<li>
<p>version</p>
</li>
<li>
<p>features (mmx, simd, rdrand, etc.) &lt;<a href="http://en.wikipedia.org/wiki/CPUID#" class="bare">http://en.wikipedia.org/wiki/CPUID#</a> EAX.3D1:_Processor_Info_and_Feature_Bits&gt;</p>
</li>
<li>
<p>caches</p>
</li>
<li>
<p>tlbs <a href="http://en.wikipedia.org/wiki/Translation_lookaside_buffer" class="bare">http://en.wikipedia.org/wiki/Translation_lookaside_buffer</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The cool thing about this instruction is that it allows you to check the CPU specs and take alternative actions based on that inside your program.</p>
</div>
<div class="paragraph">
<p>On Linux, the capacity part of this information is parsed and made available at <code>cat /proc/cpuinfo</code>. See: <a href="http://unix.stackexchange.com/questions/43539/what-do-the-flags-in-proc-cpuinfo-mean" class="bare">http://unix.stackexchange.com/questions/43539/what-do-the-flags-in-proc-cpuinfo-mean</a></p>
</div>
<div class="paragraph">
<p>There is also the <code>cpuinfo</code> command line tool that parses the CPUID instruction from the command line. Source: <a href="http://www.etallen.com/cpuid.html" class="bare">http://www.etallen.com/cpuid.html</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-x87-fpu-instructions"><a class="anchor" href="#x86-x87-fpu-instructions"></a><a class="link" href="#x86-x87-fpu-instructions">29.11. x86 x87 FPU instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.2 "X87 FPU INSTRUCTIONS"</p>
</div>
<div class="paragraph">
<p>Old floating point unit that you should likely not use anymore, prefer instead the newer <a href="#x86-simd">x86 SIMD</a> instructions.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>FPU basic examples, start here</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fadd.S">userland/arch/x86_64/fadd.S</a> FADD. The x76 FPU works on a stack of floating point numbers.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/faddp.S">userland/arch/x86_64/faddp.S</a> FADDP. Instructions with the P suffix also Pop the stack. This is often what you want for most computations, where the intermediate results don&#8217;t matter.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fldl_literal.S">userland/arch/x86_64/fldl_literal.S</a> FLDL literal. It does not seem possible to either <a href="https://stackoverflow.com/questions/6514537/how-do-i-specify-immediate-floating-point-numbers-with-inline-assembly" class="bare">https://stackoverflow.com/questions/6514537/how-do-i-specify-immediate-floating-point-numbers-with-inline-assembly</a></p>
<div class="ulist">
<ul>
<li>
<p>load floating point immediates into x86 x87 FPU registers</p>
</li>
<li>
<p>encode floating point literals in x86 instructions, including MOV</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>Bulk instructions</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fabs.S">userland/arch/x86_64/fabs.S</a> FABS: absolute value: <code>ST0 = |ST0|</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fchs.S">userland/arch/x86_64/fchs.S</a> FCHS: change sign: <code>ST0 = -ST0</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fild.S">userland/arch/x86_64/fild.S</a> FILD: Integer Load. Convert integer to float.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fld1.S">userland/arch/x86_64/fld1.S</a> FLD1: Push 1.0 to ST0. CISC!</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fldz.S">userland/arch/x86_64/fldz.S</a> FLDZ: Push 0.0 to ST0.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fscale.S">userland/arch/x86_64/fscale.S</a> FSCALE: <code>ST0 = ST0 * 2 ^ RoundTowardZero(ST1)</code></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fsqrt.S">userland/arch/x86_64/fsqrt.S</a> FSQRT: square root</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/fxch.S">userland/arch/x86_64/fxch.S</a> FXCH: swap ST0 and another register</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The ST0-ST7 x87 FPU registers are actually 80-bits wide, this can be seen from GDB with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>i r st0 st1</pre>
</div>
</div>
<div class="paragraph">
<p>By counting the number of hex digits, we have 20 digits instead of 16!</p>
</div>
<div class="paragraph">
<p>Instructions such as FLDL convert standard <a href="#ieee-754">IEEE 754</a> 64-bit values from memory into this custom 80-bit format.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/3206101/extended-80-bit-double-floating-point-in-x87-not-sse2-we-dont-miss-it" class="bare">https://stackoverflow.com/questions/3206101/extended-80-bit-double-floating-point-in-x87-not-sse2-we-dont-miss-it</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Extended_precision#x86_extended_precision_format" class="bare">https://en.wikipedia.org/wiki/Extended_precision#x86_extended_precision_format</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="x86-x87-fpu-vs-simd"><a class="anchor" href="#x86-x87-fpu-vs-simd"></a><a class="link" href="#x86-x87-fpu-vs-simd">29.11.1. x86 x87 FPU vs SIMD</a></h4>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/1844669/benefits-of-x87-over-sse" class="bare">https://stackoverflow.com/questions/1844669/benefits-of-x87-over-sse</a></p>
</div>
<div class="paragraph">
<p>Modern x86 has two main ways of doing floating point operations:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#x86-x87-fpu-instructions">x86 x87 FPU instructions</a></p>
</li>
<li>
<p><a href="#x86-simd">x86 SIMD</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Advantages of FPU:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>present in old CPUs, while SSE2 is only required in x86-64</p>
</li>
<li>
<p>contains some instructions no present in SSE, e.g. trigonometric</p>
</li>
<li>
<p>higher precision: FPU holds 80 bit Intel extension, while SSE2 only does up to 64 bit operations despite having the 128-bit register</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In GCC, you can choose between them with <code>-mfpmath=</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-simd"><a class="anchor" href="#x86-simd"></a><a class="link" href="#x86-simd">29.12. x86 SIMD</a></h3>
<div class="paragraph">
<p>Parent section: <a href="#simd-assembly">Section 28.3, &#8220;SIMD assembly&#8221;</a></p>
</div>
<div class="paragraph">
<p>History:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/MMX_(instruction_set)">MMX</a>: MultiMedia eXtension (unofficial name). 1997. MM0-MM7 64-bit registers.</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions">SSE</a>: Streaming SIMD Extensions. 1999. XMM0-XMM7 128-bit registers, XMM0-XMM15 for AMD in 64-bit mode.</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/SSE2">SSE2</a>: 2004</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/SSE3">SSE3</a>: 2006</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/SSE4">SSE4</a>: 2006</p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/Advanced_Vector_Extensions">AVX</a>: Advanced Vector Extensions. 2011. YMM0–YMM15 256-bit registers in 64-bit mode. Extension of XMM.</p>
</li>
<li>
<p>AVX2:2013</p>
</li>
<li>
<p>AVX-512: 2016. 512-bit ZMM registers. Extension of YMM.</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="x86-sse-instructions"><a class="anchor" href="#x86-sse-instructions"></a><a class="link" href="#x86-sse-instructions">29.12.1. x86 SSE instructions</a></h4>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.5 "SSE INSTRUCTIONS"</p>
</div>
<div class="sect4">
<h5 id="x86-sse-data-transfer-instructions"><a class="anchor" href="#x86-sse-data-transfer-instructions"></a><a class="link" href="#x86-sse-data-transfer-instructions">29.12.1.1. x86 SSE data transfer instructions</a></h5>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.5.1.1 "SSE Data Transfer Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movaps.S">userland/arch/x86_64/movaps.S</a>: MOVAPS: move 4 x 32-bits between two XMM registeres or XMM registers and 16-byte aligned memory</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movaps.S">userland/arch/x86_64/movaps.S</a>: MOVUPS: like MOVAPS but also works for unaligned memory</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/movss.S">userland/arch/x86_64/movss.S</a>: MOVSS: move 32-bits between two XMM registeres or XMM registers and memory</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="x86-sse-packed-arithmetic-instructions"><a class="anchor" href="#x86-sse-packed-arithmetic-instructions"></a><a class="link" href="#x86-sse-packed-arithmetic-instructions">29.12.1.2. x86 SSE packed arithmetic instructions</a></h5>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.5.1.2 "SSE Packed Arithmetic Instructions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/addpd.S">userland/arch/x86_64/addpd.S</a>: ADDPS, ADDPD: good first instruction to learn <a href="#simd-assembly">SIMD assembly</a>.</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="x86-sse-conversion-instructions"><a class="anchor" href="#x86-sse-conversion-instructions"></a><a class="link" href="#x86-sse-conversion-instructions">29.12.1.3. x86 SSE conversion instructions</a></h5>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.5.1.6 "SSE Conversion Instructions"</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="x86-sse2-instructions"><a class="anchor" href="#x86-sse2-instructions"></a><a class="link" href="#x86-sse2-instructions">29.12.2. x86 SSE2 instructions</a></h4>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.6 "SSE2 INSTRUCTIONS"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/cvttss2si.S">userland/arch/x86_64/cvttss2si.S</a>: CVTTSS2SI: convert 32-bit floating point to 32-bit integer, store the result in a general purpose register. Round towards 0.</p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="x86-paddq-instruction"><a class="anchor" href="#x86-paddq-instruction"></a><a class="link" href="#x86-paddq-instruction">29.12.2.1. x86 PADDQ instruction</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/paddq.S">userland/arch/x86_64/paddq.S</a>: PADDQ, PADDL, PADDW, PADDB</p>
</div>
<div class="paragraph">
<p>Good first instruction to learn <a href="#simd-assembly">SIMD assembly</a>.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="x86-fma"><a class="anchor" href="#x86-fma"></a><a class="link" href="#x86-fma">29.12.3. x86 fused multiply add (FMA)</a></h4>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.15 "FUSED-MULTIPLY-ADD (FMA)"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/vfmadd132pd.S">userland/arch/x86_64/vfmadd132pd.S</a>: VFMADD132PD: "Multiply packed double-precision floating-point values from xmm1 and xmm3/mem, add to xmm2 and put result in xmm1." TODO: but I don&#8217;t understand the manual, experimentally on <a href="#p51">2017 Lenovo ThinkPad P51</a> Ubuntu 19.04 host the result is stored in XMM2!</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These instructions were not part of any SSEn set: they actually have a dedicated CPUID flag for it! It appears under <code>/proc/cpuinfo</code> as <code>fma</code>. They were introduced into AVX512F however.</p>
</div>
<div class="paragraph">
<p>They are also unusual for x86 instructions in that they take 3 operands, as you would intuitively expect from the definition of FMA.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-system-instructions"><a class="anchor" href="#x86-system-instructions"></a><a class="link" href="#x86-system-instructions">29.13. x86 system instructions</a></h3>
<div class="paragraph">
<p><a href="#intel-manual-1">Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a> 5.20 "SYSTEM INSTRUCTIONS"</p>
</div>
<div class="sect3">
<h4 id="x86-rdtsc-instruction"><a class="anchor" href="#x86-rdtsc-instruction"></a><a class="link" href="#x86-rdtsc-instruction">29.13.1. x86 RDTSC instruction</a></h4>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rdtsc.S">userland/arch/x86_64/rdtsc.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/intrinsics/rdtsc.c">userland/arch/x86_64/intrinsics/rdtsc.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Try running the programs multiple times, and watch the value increase, and then try to correlate it with <code>/proc/cpuinfo</code> frequency!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>while true; do sleep 1 &amp;&amp; ./userland/arch/x86_64/rdtsc.out; done</pre>
</div>
</div>
<div class="paragraph">
<p>RDTSC stores its output to EDX:EAX, even in 64-bit mode, top bits are zeroed out.</p>
</div>
<div class="paragraph">
<p>TODO: review this section, make a more controlled userland experiment with <a href="#m5ops">m5ops</a> instrumentation.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s have some fun and try to correlate the <a href="#gem5-m5out-stats-txt-file">gem5 m5out/stats.txt file</a> <code>system.cpu.numCycles</code> cycle count with the <a href="https://en.wikipedia.org/wiki/Time_Stamp_Counter">x86 RDTSC instruction</a> that is supposed to do the same thing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland userland/arch/x86_64/inline_asm/rdtsc.S
./run --eval './arch/x86_64/rdtsc.out;m5 exit;' --emulator gem5
./gem5-stat</pre>
</div>
</div>
<div class="paragraph">
<p>RDTSC outputs a cycle count which we compare with gem5&#8217;s <code>gem5-stat</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>3828578153</code>: RDTSC</p>
</li>
<li>
<p><code>3830832635</code>: <code>gem5-stat</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>which gives pretty close results, and serve as a nice sanity check that the cycle counter is coherent.</p>
</div>
<div class="paragraph">
<p>It is also nice to see that RDTSC is a bit smaller than the <code>stats.txt</code> value, since the latter also includes the exec syscall for <code>m5</code>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/Time_Stamp_Counter" class="bare">https://en.wikipedia.org/wiki/Time_Stamp_Counter</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/13772567/how-to-get-the-cpu-cycle-count-in-x86-64-from-c" class="bare">https://stackoverflow.com/questions/13772567/how-to-get-the-cpu-cycle-count-in-x86-64-from-c</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/9887839/clock-cycle-count-wth-gcc/9887979" class="bare">https://stackoverflow.com/questions/9887839/clock-cycle-count-wth-gcc/9887979</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="x86-rdtscp-instruction"><a class="anchor" href="#x86-rdtscp-instruction"></a><a class="link" href="#x86-rdtscp-instruction">29.13.1.1. x86 RDTSCP instruction</a></h5>
<div class="paragraph">
<p>RDTSCP is like RDTSP, but it also stores the CPU ID into ECX: this is convenient because the value of RDTSC depends on which core we are currently on, so you often also want the core ID when you want the RDTSC.</p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/rdtscp.S">userland/arch/x86_64/rdtscp.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/intrinsics/rdtscp.c">userland/arch/x86_64/intrinsics/rdtscp.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We can observe its operation with the good and old <code>taskset</code>, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>taskset -c 0 ./userland/arch/x86_64/rdtscp.out | tail -n 1
taskset -c 1 ./userland/arch/x86_64/rdtscp.out | tail -n 1</pre>
</div>
</div>
<div class="paragraph">
<p>produces:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x00000000
0x00000001</pre>
</div>
</div>
<div class="paragraph">
<p>There is also the RDPID instruction that reads just the processor ID, but it appears to be very new for QEMU 4.0.0 or <a href="#p51">2017 Lenovo ThinkPad P51</a>, as it fails with SIGILL on both.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ARM has an analogous <a href="#arm-pmccntr-register">ARM PMCCNTR register</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/22310028/is-there-an-x86-instruction-to-tell-which-core-the-instruction-is-being-run-on/56622112#56622112" class="bare">https://stackoverflow.com/questions/22310028/is-there-an-x86-instruction-to-tell-which-core-the-instruction-is-being-run-on/56622112#56622112</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-thread-synchronization-primitives"><a class="anchor" href="#x86-thread-synchronization-primitives"></a><a class="link" href="#x86-thread-synchronization-primitives">29.14. x86 thread synchronization primitives</a></h3>
<div class="sect3">
<h4 id="x86-lock-prefix"><a class="anchor" href="#x86-lock-prefix"></a><a class="link" href="#x86-lock-prefix">29.14.1. x86 LOCK prefix</a></h4>
<div class="paragraph">
<p>Inline assembly example at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/x86_64_lock_inc.cpp">userland/cpp/atomic/x86_64_lock_inc.cpp</a>, see also: <a href="#atomic-cpp">atomic.cpp</a>.</p>
</div>
<div class="paragraph">
<p>Ensures that memory modifications are visible across all CPUs, which is fundamental for thread synchronization.</p>
</div>
<div class="paragraph">
<p>Apparently already automatically implied by some of the <a href="#x86-exchange-instructions">x86 exchange instructions</a></p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/8891067/what-does-the-lock-instruction-mean-in-x86-assembly/56803909#56803909" class="bare">https://stackoverflow.com/questions/8891067/what-does-the-lock-instruction-mean-in-x86-assembly/56803909#56803909</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438" class="bare">https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="x86-assembly-bibliography"><a class="anchor" href="#x86-assembly-bibliography"></a><a class="link" href="#x86-assembly-bibliography">29.15. x86 assembly bibliography</a></h3>
<div class="sect3">
<h4 id="x86-official-bibliography"><a class="anchor" href="#x86-official-bibliography"></a><a class="link" href="#x86-official-bibliography">29.15.1. x86 official bibliography</a></h4>
<div class="sect4">
<h5 id="intel-manual"><a class="anchor" href="#intel-manual"></a><a class="link" href="#intel-manual">29.15.1.1. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals</a></h5>
<div class="paragraph">
<p>We are using the May 2019 version unless otherwise noted.</p>
</div>
<div class="paragraph">
<p>There are a few download forms at: <a href="https://software.intel.com/en-us/articles/intel-sdm" class="bare">https://software.intel.com/en-us/articles/intel-sdm</a></p>
</div>
<div class="paragraph">
<p>The single PDF one is useless however because it does not have a unified ToC nor inter Volume links, so I just download the 4-part one.</p>
</div>
<div class="paragraph">
<p>The Volumes are well split, so it is usually easy to guess where you should look into.</p>
</div>
<div class="paragraph">
<p>Also I can&#8217;t find older versions on the website easily, so I just web archive everything.</p>
</div>
<div class="sect5">
<h6 id="intel-manual-1"><a class="anchor" href="#intel-manual-1"></a><a class="link" href="#intel-manual-1">29.15.1.1.1. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 1</a></h6>
<div class="paragraph">
<p>Userland basics: <a href="http://web.archive.org/web/20190606075544/https://software.intel.com/sites/default/files/managed/a4/60/253665-sdm-vol-1.pdf" class="bare">http://web.archive.org/web/20190606075544/https://software.intel.com/sites/default/files/managed/a4/60/253665-sdm-vol-1.pdf</a></p>
</div>
</div>
<div class="sect5">
<h6 id="intel-manual-2"><a class="anchor" href="#intel-manual-2"></a><a class="link" href="#intel-manual-2">29.15.1.1.2. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 2</a></h6>
<div class="paragraph">
<p>Instruction list: <a href="http://web.archive.org/web/20190606075330/https://software.intel.com/sites/default/files/managed/a4/60/325383-sdm-vol-2abcd.pdf" class="bare">http://web.archive.org/web/20190606075330/https://software.intel.com/sites/default/files/managed/a4/60/325383-sdm-vol-2abcd.pdf</a></p>
</div>
</div>
<div class="sect5">
<h6 id="intel-manual-3"><a class="anchor" href="#intel-manual-3"></a><a class="link" href="#intel-manual-3">29.15.1.1.3. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 3</a></h6>
<div class="paragraph">
<p>Kernel land: <a href="http://web.archive.org/web/20190606075534/https://software.intel.com/sites/default/files/managed/a4/60/325384-sdm-vol-3abcd.pdf" class="bare">http://web.archive.org/web/20190606075534/https://software.intel.com/sites/default/files/managed/a4/60/325384-sdm-vol-3abcd.pdf</a></p>
</div>
</div>
<div class="sect5">
<h6 id="intel-manual-4"><a class="anchor" href="#intel-manual-4"></a><a class="link" href="#intel-manual-4">29.15.1.1.4. Intel 64 and IA-32 Architectures Software Developer&#8217;s Manuals Volume 4</a></h6>
<div class="paragraph">
<p>Model specific extensions: <a href="http://web.archive.org/web/20190606075325/https://software.intel.com/sites/default/files/managed/22/0d/335592-sdm-vol-4.pdf" class="bare">http://web.archive.org/web/20190606075325/https://software.intel.com/sites/default/files/managed/22/0d/335592-sdm-vol-4.pdf</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="arm-userland-assembly"><a class="anchor" href="#arm-userland-assembly"></a><a class="link" href="#arm-userland-assembly">30. ARM userland assembly</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Arch general getting started at: <a href="#userland-assembly">Section 28, &#8220;Userland assembly&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Instructions here loosely grouped based on that of the <a href="#armarm7">ARMv7 architecture reference manual</a> Chapter A4 "The Instruction Sets".</p>
</div>
<div class="paragraph">
<p>We cover here mostly ARMv7, and then treat aarch64 differentially, since much of the ARMv7 userland is the same in aarch32.</p>
</div>
<div class="sect2">
<h3 id="introduction-to-the-arm-architecture"><a class="anchor" href="#introduction-to-the-arm-architecture"></a><a class="link" href="#introduction-to-the-arm-architecture">30.1. Introduction to the ARM architecture</a></h3>
<div class="paragraph">
<p>The <a href="https://en.wikipedia.org/wiki/ARM_architecture">ARM architecture</a> is has been used on the vast majority of mobile phones in the 2010&#8217;s, and on a large fraction of micro controllers.</p>
</div>
<div class="paragraph">
<p>It competes with <a href="#x86-userland-assembly">x86 userland assembly</a> because its implementations are designed for low power consumption, which is a major requirement of the cell phone market.</p>
</div>
<div class="paragraph">
<p>ARM is generally considered a RISC instruction set, although there are some more complex instructions which would not generally be classified as purely RISC.</p>
</div>
<div class="paragraph">
<p>ARM is developed by the British funded company ARM Holdings: <a href="https://en.wikipedia.org/wiki/Arm_Holdings" class="bare">https://en.wikipedia.org/wiki/Arm_Holdings</a> which originated as a joint venture between Acorn Computers, Apple and VLSI Technology in 1990.</p>
</div>
<div class="paragraph">
<p>ARM Holdings was bought by the Japanese giant SoftBank in 2016.</p>
</div>
<div class="sect3">
<h4 id="armv8-vs-armv7-vs-aarch64-vs-aarch32"><a class="anchor" href="#armv8-vs-armv7-vs-aarch64-vs-aarch32"></a><a class="link" href="#armv8-vs-armv7-vs-aarch64-vs-aarch32">30.1.1. ARMv8 vs ARMv7 vs AArch64 vs AArch32</a></h4>
<div class="paragraph">
<p>ARMv7 is the older architecture described at: <a href="#armarm7">ARMv7 architecture reference manual</a>.</p>
</div>
<div class="paragraph">
<p>ARMv8 is the newer architecture ISA <a href="https://developer.arm.com/docs/den0024/latest/preface">released in 2013</a> and described at: <a href="#armarm8">ARMv8 architecture reference manual</a>. It can be in either of two states:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#aarch32">AArch32</a></p>
</li>
<li>
<p>aarch64</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>In the lose terminology of this repository:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>arm</code> means basically AArch32</p>
</li>
<li>
<p><code>aarch64</code> means ARMv8 AArch64</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>ARMv8 has <a href="https://en.wikipedia.org/wiki/ARM_architecture#ARMv8-A">had several updates</a> since its release:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>v8.1: 2014</p>
</li>
<li>
<p>v8.2: 2016</p>
</li>
<li>
<p>v8.3: 2016</p>
</li>
<li>
<p>v8.4: TODO</p>
</li>
<li>
<p>v8.5: 2018</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>They are described at: <a href="#armarm8">ARMv8 architecture reference manual</a> A1.7 "ARMv8 architecture extensions".</p>
</div>
<div class="sect4">
<h5 id="aarch32"><a class="anchor" href="#aarch32"></a><a class="link" href="#aarch32">30.1.1.1. AArch32</a></h5>
<div class="paragraph">
<p>32-bit mode of operation of ARMv8.</p>
</div>
<div class="paragraph">
<p>Userland is highly / fully backwards compatible with ARMv7:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/42972096/armv8-backward-compatibility-with-armv7-snapdragon-820-vs-cortex-a15" class="bare">https://stackoverflow.com/questions/42972096/armv8-backward-compatibility-with-armv7-snapdragon-820-vs-cortex-a15</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/31848185/does-armv8-aarch32-mode-has-backward-compatible-with-armv4-armv5-or-armv6" class="bare">https://stackoverflow.com/questions/31848185/does-armv8-aarch32-mode-has-backward-compatible-with-armv4-armv5-or-armv6</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For this reason, QEMU and GAS seems to enable both AArch32 and ARMv7 under <code>arm</code> rather than <code>aarch64</code>.</p>
</div>
<div class="paragraph">
<p>There are however some extensions over ARMv7, many of them are functionality that ARMv8 has and that designers decided to backport on AArch32 as well, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#armv8-aarch32-vcvta-instruction">ARMv8 AArch32 VCVTA instruction</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="aarch32-vs-aarch64"><a class="anchor" href="#aarch32-vs-aarch64"></a><a class="link" href="#aarch32-vs-aarch64">30.1.1.2. AArch32 vs AArch64</a></h5>
<div class="paragraph">
<p>A great summary of differences can be found at: <a href="https://en.wikipedia.org/wiki/ARM_architecture#AArch64_features" class="bare">https://en.wikipedia.org/wiki/ARM_architecture#AArch64_features</a></p>
</div>
<div class="paragraph">
<p>Some random ones:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>aarch32 has two encodings: Thumb and ARM: <a href="#arm-instruction-encodings">Section 30.1.3, &#8220;ARM instruction encodings&#8221;</a></p>
</li>
<li>
<p>in ARMv8, the stack can be enforced to 16-byte alignment: <a href="#armv8-aarch64-stack-alignment">Section 30.3.2.2.1, &#8220;ARMV8 aarch64 stack alignment&#8221;</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="free-arm-implementations"><a class="anchor" href="#free-arm-implementations"></a><a class="link" href="#free-arm-implementations">30.1.2. Free ARM implementations</a></h4>
<div class="paragraph">
<p>The ARM instruction set is itself protected by patents / copyright / whatever, and you have to pay ARM Holdings a licence to implement it, even if you are creating your own custom Verilog code.</p>
</div>
<div class="paragraph">
<p>ARM has already sued people in the past for implementing ARM ISA: <a href="http://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1287452" class="bare">http://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1287452</a></p>
</div>
<div class="paragraph">
<p><a href="http://semiengineering.com/an-alternative-to-x86-arm-architectures/" class="bare">http://semiengineering.com/an-alternative-to-x86-arm-architectures/</a> mentions that:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Asanovic joked that the shortest unit of time is not the moment between a traffic light turning green in New York City and the cab driver behind the first vehicle blowing the horn; it’s someone announcing that they have created an open-source, ARM-compatible core and receiving a “cease and desist” letter from a law firm representing ARM.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>This licensing however does have the following fairness to it: ARM Holdings invents a lot of money in making a great open source software environment for the ARM ISA, so it is only natural that it should be able to get some money from hardware manufacturers for using their ISA.</p>
</div>
<div class="paragraph">
<p>Patents for very old ISAs however have expired, Amber is one implementation of those: <a href="https://en.wikipedia.org/wiki/Amber_%28processor_core%29" class="bare">https://en.wikipedia.org/wiki/Amber_%28processor_core%29</a> TODO does it have any application?</p>
</div>
<div class="paragraph">
<p>Generally, it is mostly large companies that implement the CPUs themselves. For example, the <a href="https://en.wikipedia.org/wiki/Apple_A12">Apple A12 chip</a>, which is used in iPhones, has verilog designs:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The A12 features an Apple-designed 64-bit ARMv8.3-A six-core CPU, with two high-performance cores running at 2.49 GHz called Vortex and four energy-efficient cores called Tempest.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>ARM designed CPUs however are mostly called <code>Coretx-A&lt;id&gt;</code>: <a href="https://en.wikipedia.org/wiki/List_of_applications_of_ARM_cores" class="bare">https://en.wikipedia.org/wiki/List_of_applications_of_ARM_cores</a> Vortex and Tempest are Apple designed ones.
Bibliography: <a href="https://www.quora.com/Why-is-it-that-you-need-a-license-from-ARM-to-design-an-ARM-CPU-How-are-the-instruction-sets-protected" class="bare">https://www.quora.com/Why-is-it-that-you-need-a-license-from-ARM-to-design-an-ARM-CPU-How-are-the-instruction-sets-protected</a></p>
</div>
</div>
<div class="sect3">
<h4 id="arm-instruction-encodings"><a class="anchor" href="#arm-instruction-encodings"></a><a class="link" href="#arm-instruction-encodings">30.1.3. ARM instruction encodings</a></h4>
<div class="paragraph">
<p>Understanding the basics of instruction encodings is fundamental to help you to remember what instructions do and why some things are possible or not, notably the <a href="#arm-ldr-pseudo-instruction">ARM LDR pseudo-instruction</a> and the <a href="#arm-adr-instruction">ADRP instruction</a>.</p>
</div>
<div class="paragraph">
<p>aarch32 has two "instruction sets", which to look just like encodings.</p>
</div>
<div class="paragraph">
<p>The encodings are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A32: every instruction is 4 bytes long. Can encode every instruction.</p>
</li>
<li>
<p>T32: most common instructions are 2 bytes long. Many others less common ones are 4 bytes long.</p>
<div class="paragraph">
<p>T stands for "Thumb", which is the original name for the technology, <a href="#armarm8">ARMv8 architecture reference manual</a> A1.3.2 "The ARM instruction sets" says:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>In previous documentation, these instruction sets were called the ARM and Thumb instruction sets</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>See also: <a href="#armarm8">ARMv8 architecture reference manual</a> F2.1.3 "Instruction encodings".</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Within each instruction set, there can be multiple encodings for a given function, and they are noted simply as:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A1, A2, &#8230;&#8203;: A32 encodings</p>
</li>
<li>
<p>T1, T2, ..m: T32 encodings</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The state bit <code>PSTATE.T</code> determines if the processor is in thumb mode or not. <a href="#armarm8">ARMv8 architecture reference manual</a> says that this bit it can only be read from <a href="#arm-bx-instruction">ARM BX instruction</a></p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/22660025/how-can-i-tell-if-i-am-in-arm-mode-or-thumb-mode-in-gdb" class="bare">https://stackoverflow.com/questions/22660025/how-can-i-tell-if-i-am-in-arm-mode-or-thumb-mode-in-gdb</a></p>
</div>
<div class="paragraph">
<p>TODO: details: <a href="https://stackoverflow.com/questions/22660025/how-can-i-tell-if-i-am-in-arm-mode-or-thumb-mode-in-gdb" class="bare">https://stackoverflow.com/questions/22660025/how-can-i-tell-if-i-am-in-arm-mode-or-thumb-mode-in-gdb</a> says it is <code>0x20 &amp; CPSR</code>.</p>
</div>
<div class="paragraph">
<p>This RISC-y mostly fixed instruction length design likely makes processor design easier and allows for certain optimizations, at the cost of slightly more complex assembly, as you can&#8217;t encode 4 / 8 byte addresses in a single instruction. Totally worth it IMHO.</p>
</div>
<div class="paragraph">
<p>This design can be contrasted with x86, which has widely variable instruction length.</p>
</div>
<div class="paragraph">
<p>We can swap between A32 and T32 with the BX and BLX instructions: <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.kui0100a/armasm_cihfddaf.htm" class="bare">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.kui0100a/armasm_cihfddaf.htm</a> puts it really nicely:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="ulist">
<ul>
<li>
<p>The BL and BLX instructions copy the address of the next instruction into lr (r14, the link register).</p>
</li>
<li>
<p>The BX and BLX instructions can change the processor state from ARM to Thumb, or from Thumb to ARM.</p>
<div class="ulist">
<ul>
<li>
<p>BLX label always changes the state.</p>
</li>
<li>
<p>BX Rm and BLX Rm derive the target state from bit[0] of Rm:</p>
<div class="ulist">
<ul>
<li>
<p>if bit[0] of Rm is 0, the processor changes to, or remains in, ARM state</p>
</li>
<li>
<p>if bit[0] of Rm is 1, the processor changes to, or remains in, Thumb state.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The BXJ instruction changes the processor state to Jazelle.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/28669905/what-is-the-difference-between-the-arm-thumb-and-thumb-2-instruction-encodings" class="bare">https://stackoverflow.com/questions/28669905/what-is-the-difference-between-the-arm-thumb-and-thumb-2-instruction-encodings</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="arm-thumb-encoding"><a class="anchor" href="#arm-thumb-encoding"></a><a class="link" href="#arm-thumb-encoding">30.1.3.1. ARM Thumb encoding</a></h5>
<div class="paragraph">
<p>Thumb examples are available at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/thumb.S">userland/arch/arm/thumb.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/freestanding/linux/hello_thumb.S">userland/arch/arm/freestanding/linux/hello_thumb.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For both of them, we can check that we are in thumb from inside GDB with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>disassemble</code>, and observe that some of the instructions are only 2 bytes long instead of always 4 as in ARM</p>
</li>
<li>
<p><code>print $cpsr &amp; 0x20</code> which is <code>1</code> on thumb and <code>0</code> otherwise</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>You should contrast those examples with similar non-thumb ones of course.</p>
</div>
<div class="paragraph">
<p>We also note that thumbness of those sources is determined solely by the <code>.thumb_func</code> directive, which implies that there must be some metadata to allow the linker to decide how that code should be called:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>for the freestanding example, this is determined by the first bit of the entry address ELF header as mentioned at: <a href="https://stackoverflow.com/questions/20369440/can-start-be-the-thumb-function/20374451#20374451" class="bare">https://stackoverflow.com/questions/20369440/can-start-be-the-thumb-function/20374451#20374451</a></p>
<div class="paragraph">
<p>We verify that with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch arm readelf -- -h "$(./getvar --arch arm userland_build_dir)/arch/arm/freestanding/linux/hello_thumb.out"</pre>
</div>
</div>
<div class="paragraph">
<p>The Linux kernel must use that to decide put the CPU in thumb mode: that could be done simply with a regular BX.</p>
</div>
</li>
<li>
<p>on the non-freestanding one, the linker uses some ELF metadata to decide that <code>main</code> is thumb and jumps to it appropriately: <a href="https://reverseengineering.stackexchange.com/questions/6080/how-to-detect-thumb-mode-in-arm-disassembly" class="bare">https://reverseengineering.stackexchange.com/questions/6080/how-to-detect-thumb-mode-in-arm-disassembly</a></p>
<div class="paragraph">
<p>TODO details. Does the linker then resolve thumbness with address relocation? Doesn&#8217;t this imply that the compiler cannot generate BL (never changes) or BLX (always changes) across object files, only BX (target state controlled by lower bit)?</p>
</div>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="arm-big-endian-mode"><a class="anchor" href="#arm-big-endian-mode"></a><a class="link" href="#arm-big-endian-mode">30.1.3.2. ARM big endian mode</a></h5>
<div class="paragraph">
<p>ARM can switch between big and little endian mode on the fly!</p>
</div>
<div class="paragraph">
<p>However, everyone only uses little endian, so the big endian ecosystem is not as supported.</p>
</div>
<div class="paragraph">
<p>TODO is there any advantage of using big endian?</p>
</div>
<div class="paragraph">
<p>Here Peter mentions that QEMU does "support" big endian in theory, but that there are no machines for it not sure what that implies: <a href="https://stackoverflow.com/questions/41571643/emulatin-big-endian-arm-system-with-qemu" class="bare">https://stackoverflow.com/questions/41571643/emulatin-big-endian-arm-system-with-qemu</a></p>
</div>
<div class="paragraph">
<p>We can try it out quickly in user mode with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>touch userland/arch/aarch64/freestanding/linux/hello.S
./build-userland --arch aarch64 --ccflags=-mbig-endian userland/arch/aarch64/freestanding/linux/hello.S
./run --arch aarch64 --userland userland/arch/aarch64/freestanding/linux/hello.S</pre>
</div>
</div>
<div class="paragraph">
<p>and it fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Invalid ELF image for this architecture</pre>
</div>
</div>
<div class="paragraph">
<p>From this we can guess that the big endian metadata is actually stored in the <a href="#elf">ELF</a> file, and confirm that with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain \
  --arch aarch64 \
  readelf \
  -- \
  --file-header "$(./getvar --arch aarch64 userland_build_dir)/arch/aarch64/freestanding/linux/hello.out" \
;</pre>
</div>
</div>
<div class="paragraph">
<p>which contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Data:                              2's complement, big endian</pre>
</div>
</div>
<div class="paragraph">
<p>instead of the default:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Data:                              2's complement, little endian</pre>
</div>
</div>
<div class="paragraph">
<p>TODO does the Linux kernel support running big endian executables? I tried after building the big endian executable:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch aarch64
./run --arch aarch64 --eval-after ./arch/aarch64/freestanding/linux/hello.out</pre>
</div>
</div>
<div class="paragraph">
<p>but that failed with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/lkmc/arch/aarch64/freestanding/linux/hello.out: line 1: ELF@x@0@8@: not found
/lkmc/arch/aarch64/freestanding/linux/hello.out: line 2: @@: not found
/lkmc/arch/aarch64/freestanding/linux/hello.out: line 3: syntax error: unexpected ")"</pre>
</div>
</div>
<div class="paragraph">
<p>TODO:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>can you compile the Linux kernel itself as big endian? Looks like yes since there is a <a href="https://github.com/torvalds/linux/blob/v5.1/arch/arm64/Kconfig#L791"><code>config CPU_BIG_ENDIAN</code></a> See also: <a href="https://unix.stackexchange.com/questions/378829/getting-big-endian-linux-build-to-boot-on-arm-with-u-boot" class="bare">https://unix.stackexchange.com/questions/378829/getting-big-endian-linux-build-to-boot-on-arm-with-u-boot</a></p>
</li>
<li>
<p>how can be is the endianess be checked and modified in the CPU?</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-branch-instructions"><a class="anchor" href="#arm-branch-instructions"></a><a class="link" href="#arm-branch-instructions">30.2. ARM branch instructions</a></h3>
<div class="sect3">
<h4 id="arm-b-instruction"><a class="anchor" href="#arm-b-instruction"></a><a class="link" href="#arm-b-instruction">30.2.1. ARM B instruction</a></h4>
<div class="paragraph">
<p>Unconditional branch.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/b.S">userland/arch/arm/b.S</a></p>
</div>
<div class="paragraph">
<p>The encoding stores PC offsets in 24 bits. The destination must be a multiple of 4, which is easy since all instructions are 4 bytes.</p>
</div>
<div class="paragraph">
<p>This allows for 26 bit long jumps, which is 64 MiB.</p>
</div>
<div class="paragraph">
<p>TODO: what to do if we want to jump longer than that?</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-beq-instruction"><a class="anchor" href="#arm-beq-instruction"></a><a class="link" href="#arm-beq-instruction">30.2.2. ARM BEQ instruction</a></h4>
<div class="paragraph">
<p>Branch if equal based on the status registers.</p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/beq.S">userland/arch/arm/beq.S</a>.</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/beq.S">userland/arch/aarch64/beq.S</a>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The family of instructions includes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>BEQ: branch if equal</p>
</li>
<li>
<p>BNE: branch if not equal</p>
</li>
<li>
<p>BLE: less or equal</p>
</li>
<li>
<p>BGE: greater or equal</p>
</li>
<li>
<p>BLT: less than</p>
</li>
<li>
<p>BGT: greater than</p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="arm-bl-instruction"><a class="anchor" href="#arm-bl-instruction"></a><a class="link" href="#arm-bl-instruction">30.2.3. ARM BL instruction</a></h4>
<div class="paragraph">
<p>Branch with link, i.e. branch and store the return address on the RL register.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/bl.S">userland/arch/arm/bl.S</a></p>
</div>
<div class="paragraph">
<p>This is the major way to make function calls.</p>
</div>
<div class="paragraph">
<p>The current ARM / Thumb mode is encoded in the least significant bit of lr.</p>
</div>
<div class="sect4">
<h5 id="arm-bx-instruction"><a class="anchor" href="#arm-bx-instruction"></a><a class="link" href="#arm-bx-instruction">30.2.3.1. ARM BX instruction</a></h5>
<div class="paragraph">
<p>See: <a href="#arm-thumb-encoding">Section 30.1.3.1, &#8220;ARM Thumb encoding&#8221;</a></p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-ret-instruction"><a class="anchor" href="#armv8-aarch64-ret-instruction"></a><a class="link" href="#armv8-aarch64-ret-instruction">30.2.3.2. ARMv8 aarch64 ret instruction</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/ret.S">userland/arch/aarch64/ret.S</a></p>
</div>
<div class="paragraph">
<p>ARMv8 AArch64 only:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>there is no BX in AArch64 since no Thumb to worry about, so it is called just BR</p>
</li>
<li>
<p>the RET instruction was added in addition to BR, with the following differences:</p>
<div class="ulist">
<ul>
<li>
<p>provides a hint that this is a function call return</p>
</li>
<li>
<p>has a default argument X30 if none is given. This is where BL puts the return value.</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>See also: <a href="https://stackoverflow.com/questions/32304646/arm-assembly-branch-to-address-inside-register-or-memory/54145818#54145818" class="bare">https://stackoverflow.com/questions/32304646/arm-assembly-branch-to-address-inside-register-or-memory/54145818#54145818</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-cbz-instruction"><a class="anchor" href="#arm-cbz-instruction"></a><a class="link" href="#arm-cbz-instruction">30.2.4. ARM CBZ instruction</a></h4>
<div class="paragraph">
<p>Compare and branch if zero.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/cbz.S">userland/arch/aarch64/cbz.S</a></p>
</div>
<div class="paragraph">
<p>Only in ARMv8 and ARMv7 Thumb mode, not in armv7 ARM mode.</p>
</div>
<div class="paragraph">
<p>Very handy!</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-conditional-execution"><a class="anchor" href="#arm-conditional-execution"></a><a class="link" href="#arm-conditional-execution">30.2.5. ARM conditional execution</a></h4>
<div class="paragraph">
<p>Weirdly, <a href="#arm-b-instruction">ARM B instruction</a> and family are not the only instructions that can execute conditionally on the flags: the same also applies to most instructions, e.g. ADD.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/cond.S">userland/arch/arm/cond.S</a></p>
</div>
<div class="paragraph">
<p>Just add the usual <code>eq</code>, <code>ne</code>, etc. suffixes just as for B.</p>
</div>
<div class="paragraph">
<p>The list of all extensions is documented at <a href="#armarm7">ARMv7 architecture reference manual</a> "A8.3 Conditional execution".</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-load-and-store-instructions"><a class="anchor" href="#arm-load-and-store-instructions"></a><a class="link" href="#arm-load-and-store-instructions">30.3. ARM load and store instructions</a></h3>
<div class="paragraph">
<p>In ARM, there are only two instruction families that do memory access:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-ldr-instruction">ARM LDR instruction</a> to load from memory to registers</p>
</li>
<li>
<p><a href="#arm-str-instruction">ARM STR instruction</a> to store from registers to memory</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Everything else works on register and immediates.</p>
</div>
<div class="paragraph">
<p>This is part of the RISC-y beauty of the ARM instruction set, unlike x86 in which several operations can read from memory, and helps to predict how to optimize for a given CPU pipeline.</p>
</div>
<div class="paragraph">
<p>This kind of architecture is called a <a href="https://en.wikipedia.org/wiki/Load/store_architecture">Load/store architecture</a>.</p>
</div>
<div class="sect3">
<h4 id="arm-ldr-instruction"><a class="anchor" href="#arm-ldr-instruction"></a><a class="link" href="#arm-ldr-instruction">30.3.1. ARM LDR instruction</a></h4>
<div class="sect4">
<h5 id="arm-ldr-pseudo-instruction"><a class="anchor" href="#arm-ldr-pseudo-instruction"></a><a class="link" href="#arm-ldr-pseudo-instruction">30.3.1.1. ARM LDR pseudo-instruction</a></h5>
<div class="paragraph">
<p>LDR can be either a regular instruction that loads stuff into memory, or also a pseudo-instruction (assembler magic): <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0041c/Babbfdih.html" class="bare">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0041c/Babbfdih.html</a></p>
</div>
<div class="paragraph">
<p>The pseudo instruction version is when an equal sign appears on one of the operators.</p>
</div>
<div class="paragraph">
<p>The LDR pseudo instruction can automatically create hidden variables in a place called the "literal pool", and load them from memory with PC relative loads.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/ldr_pseudo.S">userland/arch/arm/ldr_pseudo.S</a></p>
</div>
<div class="paragraph">
<p>This is done basically because all instructions are 32-bit wide, and there is not enough space to encode 32-bit addresses in them.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/37840754/what-does-an-equals-sign-on-the-right-side-of-a-ldr-instruction-in-arm-mean" class="bare">https://stackoverflow.com/questions/37840754/what-does-an-equals-sign-on-the-right-side-of-a-ldr-instruction-in-arm-mean</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/17214962/what-is-the-difference-between-label-equals-sign-and-label-brackets-in-ar" class="bare">https://stackoverflow.com/questions/17214962/what-is-the-difference-between-label-equals-sign-and-label-brackets-in-ar</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly" class="bare">https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="arm-addressing-modes"><a class="anchor" href="#arm-addressing-modes"></a><a class="link" href="#arm-addressing-modes">30.3.1.2. ARM addressing modes</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/address_modes.S">userland/arch/arm/address_modes.S</a></p>
</div>
<div class="paragraph">
<p>Load and store instructions can update the source register with the following modes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>offset: add an offset, don&#8217;t change the address register. Notation:</p>
<div class="literalblock">
<div class="content">
<pre>ldr r1, [r0, 4]</pre>
</div>
</div>
</li>
<li>
<p>pre-indexed: change the address register, and then use it modified. Notation:</p>
<div class="literalblock">
<div class="content">
<pre>ldr r1, [r0, 4]!</pre>
</div>
</div>
</li>
<li>
<p>post-indexed: use the address register unmodified, and then modify it. Notation:</p>
<div class="literalblock">
<div class="content">
<pre>ldr r1, [r0], 4</pre>
</div>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The offset itself can come from the following sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>immediate</p>
</li>
<li>
<p>register</p>
</li>
<li>
<p>scaled register: left shift the register and use that as an offset</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The indexed modes are convenient to loop over arrays.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="#armarm7">ARMv7 architecture reference manual</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A4.6.5 "Addressing modes"</p>
</li>
<li>
<p>A8.5 "Memory accesses"</p>
</li>
</ul>
</div>
<div class="paragraph">
<p><a href="#armarm8">ARMv8 architecture reference manual</a>: C1.3.3 "Load/Store addressing modes"</p>
</div>
<div class="sect5">
<h6 id="arm-loop-over-array"><a class="anchor" href="#arm-loop-over-array"></a><a class="link" href="#arm-loop-over-array">30.3.1.2.1. ARM loop over array</a></h6>
<div class="paragraph">
<p>As an application of the post-indexed addressing mode, let&#8217;s increment an array.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/inc_array.S">userland/arch/arm/inc_array.S</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="arm-ldrh-and-ldrb-instructions"><a class="anchor" href="#arm-ldrh-and-ldrb-instructions"></a><a class="link" href="#arm-ldrh-and-ldrb-instructions">30.3.1.3. ARM LDRH and LDRB instructions</a></h5>
<div class="paragraph">
<p>There are LDR variants that load less than full 4 bytes:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/ldrb.S">userland/arch/arm/ldrb.S</a>: load byte</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/ldrh.S">userland/arch/arm/ldrh.S</a>: load half word</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>These also have signed and unsigned versions to either zero or one extend the result:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/ldrsw.S">userland/arch/aarch64/ldrsw.S</a>: load byte and sign extend</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-str-instruction"><a class="anchor" href="#arm-str-instruction"></a><a class="link" href="#arm-str-instruction">30.3.2. ARM STR instruction</a></h4>
<div class="paragraph">
<p>Store from memory into registers.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/str.S">userland/arch/arm/str.S</a></p>
</div>
<div class="paragraph">
<p>Basically everything that applies to <a href="#arm-ldr-instruction">ARM LDR instruction</a> also applies here so we won&#8217;t go into much detail.</p>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-str-instruction"><a class="anchor" href="#armv8-aarch64-str-instruction"></a><a class="link" href="#armv8-aarch64-str-instruction">30.3.2.1. ARMv8 aarch64 STR instruction</a></h5>
<div class="paragraph">
<p>PC-relative STR is not possible in aarch64.</p>
</div>
<div class="paragraph">
<p>For LDR it works <a href="#arm-ldr-instruction">as in aarch32</a>.</p>
</div>
<div class="paragraph">
<p>As a result, it is not possible to load from the literal pool for STR.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/str.S">userland/arch/aarch64/str.S</a></p>
</div>
<div class="paragraph">
<p>This can be seen from <a href="#armarm8">ARMv8 architecture reference manual</a> C3.2.1 "Load/Store register": LDR simply has on extra PC encoding that STR does not.</p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-ldp-and-stp-instructions"><a class="anchor" href="#armv8-aarch64-ldp-and-stp-instructions"></a><a class="link" href="#armv8-aarch64-ldp-and-stp-instructions">30.3.2.2. ARMv8 aarch64 LDP and STP instructions</a></h5>
<div class="paragraph">
<p>Push a pair of registers to the stack.</p>
</div>
<div class="paragraph">
<p>TODO minimal example. Currently used in <code>LKMC_PROLOGUE</code> at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/aarch64.h">lkmc/aarch64.h</a> since it is the main way to restore register state.</p>
</div>
<div class="sect5">
<h6 id="armv8-aarch64-stack-alignment"><a class="anchor" href="#armv8-aarch64-stack-alignment"></a><a class="link" href="#armv8-aarch64-stack-alignment">30.3.2.2.1. ARMV8 aarch64 stack alignment</a></h6>
<div class="paragraph">
<p>In ARMv8, the stack can be enforced to 16-byte alignment.</p>
</div>
<div class="paragraph">
<p>This is why the main way to push things to stack is with 8-byte pair pushes with the <a href="#armv8-aarch64-ldp-and-stp-instructions">ARMv8 aarch64 LDP and STP instructions</a>.</p>
</div>
<div class="paragraph">
<p><a href="#armarm8-db">ARMv8 architecture reference manual db</a> C1.3.3 "Load/Store addressing modes" says:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>When stack alignment checking is enabled by system software and the base register is the SP, the current stack pointer must be initially quadword aligned, that is aligned to 16 bytes. Misalignment generates a Stack Alignment fault. The offset does not have to be a multiple of 16 bytes unless the specific Load/Store instruction requires this. SP cannot be used as a register offset.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p><a href="#armarm8-db">ARMv8 architecture reference manual db</a> C3.2 "Loads and stores" says:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The additional control bits SCTLR_ELx.SA and SCTLR_EL1.SA0 control whether the stack pointer must be quadword aligned when used as a base register. See SP alignment checking on page D1-2164. Using a misaligned stack pointer generates an SP alignment fault exception.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p><a href="#armarm8-db">ARMv8 architecture reference manual db</a> D1.8.2 "SP alignment checking" is then the main section.</p>
</div>
<div class="paragraph">
<p>TODO: what does the ABI say on this? Why don&#8217;t I observe faults on QEMU as mentioned at: <a href="https://stackoverflow.com/questions/212466/what-is-a-bus-error/31877230#31877230" class="bare">https://stackoverflow.com/questions/212466/what-is-a-bus-error/31877230#31877230</a></p>
</div>
<div class="paragraph">
<p>See also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/38535738/does-aarch64-support-unaligned-access" class="bare">https://stackoverflow.com/questions/38535738/does-aarch64-support-unaligned-access</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-ldmia-instruction"><a class="anchor" href="#arm-ldmia-instruction"></a><a class="link" href="#arm-ldmia-instruction">30.3.3. ARM LDMIA instruction</a></h4>
<div class="paragraph">
<p>Pop values form stack into the register and optionally update the address register.</p>
</div>
<div class="paragraph">
<p>STMDB is the push version.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/ldmia.S">userland/arch/arm/ldmia.S</a></p>
</div>
<div class="paragraph">
<p>The mnemonics stand for:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>STMDB: STore Multiple Decrement Before</p>
</li>
<li>
<p>LDMIA: LoaD Multiple Increment After</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/push.S">userland/arch/arm/push.S</a></p>
</div>
<div class="paragraph">
<p>PUSH and POP are just mnemonics STDMDB and LDMIA using the stack pointer SP as address register:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>stmdb sp!, reglist
ldmia sp!, reglist</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>!</code> indicates that we want to update the register.</p>
</div>
<div class="paragraph">
<p>The registers are encoded as single bits inside the instruction: each bit represents one register.</p>
</div>
<div class="paragraph">
<p>As a consequence, the push order is fixed no matter how you write the assembly instruction: there is just not enough space to encode ordering.</p>
</div>
<div class="paragraph">
<p>AArch64 loses those instructions, likely because it was not possible anymore to encode all registers: <a href="https://stackoverflow.com/questions/27941220/push-lr-and-pop-lr-in-arm-arch64" class="bare">https://stackoverflow.com/questions/27941220/push-lr-and-pop-lr-in-arm-arch64</a> and replaces them with the <a href="#armv8-aarch64-ldp-and-stp-instructions">ARMv8 aarch64 LDP and STP instructions</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-data-processing-instructions"><a class="anchor" href="#arm-data-processing-instructions"></a><a class="link" href="#arm-data-processing-instructions">30.4. ARM data processing instructions</a></h3>
<div class="paragraph">
<p>Arithmetic:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/mul.S">userland/arch/arm/mul.S</a>: multiply</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/sub.S">userland/arch/arm/sub.S</a>: subtract</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/rbit.S">userland/arch/arm/rbit.S</a>: reverse bit order</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/rev.S">userland/arch/arm/rev.S</a>: reverse byte order</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/tst.S">userland/arch/arm/tst.S</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="arm-cset-instruction"><a class="anchor" href="#arm-cset-instruction"></a><a class="link" href="#arm-cset-instruction">30.4.1. ARM CSET instruction</a></h4>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/cset.S">userland/arch/aarch64/cset.S</a></p>
</div>
<div class="paragraph">
<p>Set a register conditionally depending on the condition flags:</p>
</div>
<div class="paragraph">
<p>ARMv8-only, likely because in ARMv8 you can&#8217;t have conditional suffixes for every instruction.</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-bitwise-instructions"><a class="anchor" href="#arm-bitwise-instructions"></a><a class="link" href="#arm-bitwise-instructions">30.4.2. ARM bitwise instructions</a></h4>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/and.S">userland/arch/arm/and.S</a> AND</p>
</li>
<li>
<p>EOR: exclusive OR</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/orr.S">userland/arch/arm/orr.S</a>: OR</p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/clz.S">userland/arch/arm/clz.S</a>: count leading zeroes</p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="arm-bic-instruction"><a class="anchor" href="#arm-bic-instruction"></a><a class="link" href="#arm-bic-instruction">30.4.2.1. ARM BIC instruction</a></h5>
<div class="paragraph">
<p>Bitwise Bit Clear: clear some bits.</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dest = left &amp; ~right</pre>
</div>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/bic.S">userland/arch/arm/bic.S</a></p>
</div>
</div>
<div class="sect4">
<h5 id="arm-ubfm-instruction"><a class="anchor" href="#arm-ubfm-instruction"></a><a class="link" href="#arm-ubfm-instruction">30.4.2.2. ARM UBFM instruction</a></h5>
<div class="paragraph">
<p>Unsigned Bitfield Move.</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>copies any number of low-order bits from a source register into the same number of adjacent bits at any position in the destination register, with zeros in the upper and lower bits.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/ubfm.S">userland/arch/aarch64/ubfm.S</a></p>
</div>
<div class="paragraph">
<p>TODO: explain full behaviour. Very complicated. Has several simpler to understand aliases.</p>
</div>
<div class="sect5">
<h6 id="arm-ubfx-instruction"><a class="anchor" href="#arm-ubfx-instruction"></a><a class="link" href="#arm-ubfx-instruction">30.4.2.2.1. ARM UBFX instruction</a></h6>
<div class="paragraph">
<p>Alias for:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>UBFM &lt;Wd&gt;, &lt;Wn&gt;, #&lt;lsb&gt;, #(&lt;lsb&gt;+&lt;width&gt;-1)</pre>
</div>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/ubfx.S">userland/arch/aarch64/ubfx.S</a></p>
</div>
<div class="paragraph">
<p>The operation:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>UBFX dest, src, lsb, width</pre>
</div>
</div>
<div class="paragraph">
<p>does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>dest = (src &amp; ((1 &lt;&lt; width) - 1)) &gt;&gt; lsb;</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/8366625/arm-bit-field-extract" class="bare">https://stackoverflow.com/questions/8366625/arm-bit-field-extract</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="arm-bfm-instruction"><a class="anchor" href="#arm-bfm-instruction"></a><a class="link" href="#arm-bfm-instruction">30.4.2.3. ARM BFM instruction</a></h5>
<div class="paragraph">
<p>TODO: explain. Similar to <a href="#arm-ubfm-instruction">UBFM</a> but leave untouched bits unmodified.</p>
</div>
<div class="sect5">
<h6 id="arm-bfi-instruction"><a class="anchor" href="#arm-bfi-instruction"></a><a class="link" href="#arm-bfi-instruction">30.4.2.3.1. ARM BFI instruction</a></h6>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/bfi.S">userland/arch/arm/bfi.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/bfi.S">userland/arch/aarch64/bfi.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Move the lower bits of source register into any position in the destination:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>ARMv8: an alias for <a href="#arm-bfm-instruction">ARM BFM instruction</a></p>
</li>
<li>
<p>ARMv7: a real instruction</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-mov-instruction"><a class="anchor" href="#arm-mov-instruction"></a><a class="link" href="#arm-mov-instruction">30.4.3. ARM MOV instruction</a></h4>
<div class="paragraph">
<p>Move an immediate to a register, or a register to another register.</p>
</div>
<div class="paragraph">
<p>Cannot load from or to memory, since only the LDR and STR instruction families can do that in ARM as mentioned at: <a href="#arm-load-and-store-instructions">Section 30.3, &#8220;ARM load and store instructions&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/mov.S">userland/arch/arm/mov.S</a></p>
</div>
<div class="paragraph">
<p>Since every instruction <a href="#arm-instruction-encodings">has a fixed 4 byte size</a>, there is not enough space to encode arbitrary 32-bit immediates in a single instruction, since some of the bits are needed to actually encode the instruction itself.</p>
</div>
<div class="paragraph">
<p>The solutions to this problem are mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/38689886/loading-32-bit-values-to-a-register-in-arm-assembly" class="bare">https://stackoverflow.com/questions/38689886/loading-32-bit-values-to-a-register-in-arm-assembly</a></p>
</li>
<li>
<p><a href="https://community.arm.com/processors/b/blog/posts/how-to-load-constants-in-assembly-for-arm-architecture" class="bare">https://community.arm.com/processors/b/blog/posts/how-to-load-constants-in-assembly-for-arm-architecture</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Summary of solutions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-movw-and-movt-instructions">ARM movw and movt instructions</a></p>
</li>
<li>
<p>place it in memory. But then how to load the address, which is also a 32-bit value?</p>
<div class="ulist">
<ul>
<li>
<p>use pc-relative addressing if the memory is close enough</p>
</li>
<li>
<p>use <a href="#arm-bitwise-instructions">ORR</a> encodable shifted immediates</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The blog article summarizes nicely which immediates can be encoded and the design rationale:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>An Operand 2 immediate must obey the following rule to fit in the instruction: an 8-bit value rotated right by an even number of bits between 0 and 30 (inclusive). This allows for constants such as 0xFF (0xFF rotated right by 0), 0xFF00 (0xFF rotated right by 24) or 0xF000000F (0xFF rotated right by 4).</p>
</div>
<div class="paragraph">
<p>In software - especially in languages like C - constants tend to be small. When they are not small they tend to be bit masks. Operand 2 immediates provide a reasonable compromise between constant coverage and encoding space; most common constants can be encoded directly.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Assemblers however support magic memory allocations which may hide what is truly going on: <a href="https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly" class="bare">https://stackoverflow.com/questions/14046686/why-use-ldr-over-mov-or-vice-versa-in-arm-assembly</a> Always ask your friendly disassembly for a good confirmation.</p>
</div>
<div class="sect4">
<h5 id="arm-movw-and-movt-instructions"><a class="anchor" href="#arm-movw-and-movt-instructions"></a><a class="link" href="#arm-movw-and-movt-instructions">30.4.3.1. ARM movw and movt instructions</a></h5>
<div class="paragraph">
<p>Set the higher or lower 16 bits of a register to an immediate in one go.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/movw.S">userland/arch/arm/movw.S</a></p>
</div>
<div class="paragraph">
<p>The armv8 version analogue is <a href="#armv8-aarch64-movk-instruction">ARMv8 aarch64 movk instruction</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-movk-instruction"><a class="anchor" href="#armv8-aarch64-movk-instruction"></a><a class="link" href="#armv8-aarch64-movk-instruction">30.4.3.2. ARMv8 aarch64 movk instruction</a></h5>
<div class="paragraph">
<p>Fill a 64 bit register with 4 16-bit instructions one at a time.</p>
</div>
<div class="paragraph">
<p>Similar to <a href="#arm-movw-and-movt-instructions">ARM movw and movt instructions</a> in v7.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/movk.S">userland/arch/aarch64/movk.S</a></p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/27938768/moving-a-32-bit-constant-in-arm-arch64-register" class="bare">https://stackoverflow.com/questions/27938768/moving-a-32-bit-constant-in-arm-arch64-register</a></p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-movn-instruction"><a class="anchor" href="#armv8-aarch64-movn-instruction"></a><a class="link" href="#armv8-aarch64-movn-instruction">30.4.3.3. ARMv8 aarch64 movn instruction</a></h5>
<div class="paragraph">
<p>Set 16-bits negated and the rest to <code>1</code>.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/movn.S">userland/arch/aarch64/movn.S</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-data-processing-instruction-suffixes"><a class="anchor" href="#arm-data-processing-instruction-suffixes"></a><a class="link" href="#arm-data-processing-instruction-suffixes">30.4.4. ARM data processing instruction suffixes</a></h4>
<div class="sect4">
<h5 id="arm-shift-suffixes"><a class="anchor" href="#arm-shift-suffixes"></a><a class="link" href="#arm-shift-suffixes">30.4.4.1. ARM shift suffixes</a></h5>
<div class="paragraph">
<p>Most data processing instructions can also optionally shift the second register operand.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/shift.S">userland/arch/arm/shift.S</a></p>
</div>
<div class="paragraph">
<p>The shift types are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>LSR and LFL: Logical Shift Right / Left. Insert zeroes.</p>
</li>
<li>
<p>ROR: Rotate Right / Left. Wrap bits around.</p>
</li>
<li>
<p>ASR: Arithmetic Shift Right. Keep sign.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Documented at: <a href="#armarm7">ARMv7 architecture reference manual</a> "A4.4.1 Standard data-processing instructions"</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-s-suffix"><a class="anchor" href="#arm-s-suffix"></a><a class="link" href="#arm-s-suffix">30.4.4.2. ARM S suffix</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/s_suffix.S">userland/arch/arm/s_suffix.S</a></p>
</div>
<div class="paragraph">
<p>The <code>S</code> suffix, present on most <a href="#arm-data-processing-instructions">ARM data processing instructions</a>, makes the instruction also set the Status register flags that control conditional jumps.</p>
</div>
<div class="paragraph">
<p>If the result of the operation is <code>0</code>, then it triggers BEQ, since comparison is a subtraction, with success on 0.</p>
</div>
<div class="paragraph">
<p>CMP sets the flags by default of course.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-adr-instruction"><a class="anchor" href="#arm-adr-instruction"></a><a class="link" href="#arm-adr-instruction">30.4.5. ARM ADR instruction</a></h4>
<div class="paragraph">
<p>Similar rationale to the <a href="#arm-ldr-pseudo-instruction">ARM LDR pseudo-instruction</a>, allowing to easily store a PC-relative reachable address into a register in one go, to overcome the 4-byte fixed instruction size.</p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/adr.S">userland/arch/arm/adr.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/adr.S">userland/arch/aarch64/adr.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/adrp.S">userland/arch/aarch64/adrp.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>More details: <a href="https://stackoverflow.com/questions/41906688/what-are-the-semantics-of-adrp-and-adrl-instructions-in-arm-assembly/54042899#54042899" class="bare">https://stackoverflow.com/questions/41906688/what-are-the-semantics-of-adrp-and-adrl-instructions-in-arm-assembly/54042899#54042899</a></p>
</div>
<div class="sect4">
<h5 id="arm-adrl-instruction"><a class="anchor" href="#arm-adrl-instruction"></a><a class="link" href="#arm-adrl-instruction">30.4.5.1. ARM ADRL instruction</a></h5>
<div class="paragraph">
<p>See: <a href="#arm-adr-instruction">Section 30.4.5, &#8220;ARM ADR instruction&#8221;</a>.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-miscellaneous-instructions"><a class="anchor" href="#arm-miscellaneous-instructions"></a><a class="link" href="#arm-miscellaneous-instructions">30.5. ARM miscellaneous instructions</a></h3>
<div class="sect3">
<h4 id="arm-nop-instruction"><a class="anchor" href="#arm-nop-instruction"></a><a class="link" href="#arm-nop-instruction">30.5.1. ARM NOP instruction</a></h4>
<div class="paragraph">
<p>Parent section: <a href="#nop-instructions">Section 28.10, &#8220;NOP instructions&#8221;</a></p>
</div>
<div class="paragraph">
<p>There are a few different ways to encode NOP, notably MOV a register into itself, and a dedicated miscellaneous instruction.</p>
</div>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/nop.S">userland/arch/arm/nop.S</a></p>
</div>
<div class="paragraph">
<p>Try disassembling the executable to see what the assembler is emitting:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>gdb-multiarch -batch -ex 'arch arm' -ex "file v7/nop.out" -ex "disassemble/rs asm_main_after_prologue"</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/1875491/nop-for-iphone-binaries" class="bare">https://stackoverflow.com/questions/1875491/nop-for-iphone-binaries</a></p>
</div>
</div>
<div class="sect3">
<h4 id="arm-udf-instruction"><a class="anchor" href="#arm-udf-instruction"></a><a class="link" href="#arm-udf-instruction">30.5.2. ARM UDF instruction</a></h4>
<div class="paragraph">
<p>Guaranteed undefined! Therefore raise illegal instruction signal. Used by GCC <code>__builtin_trap</code> apparently: <a href="https://stackoverflow.com/questions/16081618/programmatically-cause-undefined-instruction-exception" class="bare">https://stackoverflow.com/questions/16081618/programmatically-cause-undefined-instruction-exception</a></p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/udf.S">userland/arch/arm/udf.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/udf.S">userland/arch/aarch64/udf.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Why GNU GAS 2.29 does not have a mnemonic for it in A64 because it is very recent: shows in <a href="#armarm8-db">ARMv8 architecture reference manual db</a> but not <code>ca</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-system-register-instructions"><a class="anchor" href="#arm-system-register-instructions"></a><a class="link" href="#arm-system-register-instructions">30.5.3. ARM system register instructions</a></h4>
<div class="paragraph">
<p>Examples of using them can be found at: <a href="#dump-regs">dump_regs</a></p>
</div>
<div class="paragraph">
<p>aarch64 only uses exactly 2 instructions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>MRS: reads a system register to a regular register</p>
</li>
<li>
<p>MSR: writes to the system register</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>aarch32 is a bit more messy due to older setups, we have both:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>MRS and MSR which are much like in aarch64</p>
</li>
<li>
<p>coprocessor accesses:</p>
<div class="ulist">
<ul>
<li>
<p>MRC: reads a system register, C means coprocessor, which is how system registers were previously known as</p>
</li>
<li>
<p>MCR: write to the system register</p>
</li>
<li>
<p>MRRC: like MRC, but used for the system registers that are marked as 64-bit, and reads to two general purpose register</p>
</li>
<li>
<p>MCRR: write version of MCRR</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO why both? For example, as mentioned at <a href="https://stackoverflow.com/questions/62920281/cross-compilng-c-program-for-armv8-a-in-linux-x86-64-system/62922677#62922677" class="bare">https://stackoverflow.com/questions/62920281/cross-compilng-c-program-for-armv8-a-in-linux-x86-64-system/62922677#62922677</a> a register that was accessed with MRC in armv7 can move to MRS in aarch64, as is the case for:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mrs r0, ctr     /* aarch32 */
mrc x0, ctr_el0 /* aarch64 */</pre>
</div>
</div>
<div class="paragraph">
<p>Other functionality has moved away from coprocessors into actual instructions, e.g. cache invalidation:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* aarch32: DCISW, Data Cache line Invalidate by Set/Way. */
mcr     p15, 0, r5, c7, c6, 2

/* aarch64: moved to one of the DC instruction variants. */
dc isw</pre>
</div>
</div>
<div class="paragraph">
<p><a href="#armarm8-fa">ARMv8 architecture reference manual db</a> G1.19.4 "Background to the System register interface" says that only CP14 and CP15 are specified by the ISA:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The interface to the System registers was originally defined as part of a generic coprocessor interface, that gave access to 15 coprocessors, CP0 - CP15. Of these, CP8 - CP15 were reserved for use by Arm, while CP0 - CP7 were available for IMPLEMENTATION DEFINED coprocessors.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>and the actual coprocessor registers are specified in Chapter G7 "AArch32 System Register Encoding" at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CP14: Table G7-1 "Mapping of (coproc ==0b1110) MCR, MRC, and MRRC instruction arguments to System registers"</p>
</li>
<li>
<p>CP15: Table G7-3 "VMSAv8-32 (coproc==0b1111) register summary, in MCR/MRC parameter order."</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The actual MRC assembly does not exactly match the order of that table, this is how you can decode it, sample MCR:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mcr     p15, 0, r5, c7, c6, 2</pre>
</div>
</div>
<div class="paragraph">
<p>what each part means:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mcr     p&lt;coproc&gt;, &lt;opc1&gt;, &lt;src-dest-reg&gt;, &lt;CRn&gt;, &lt;CRm&gt;, &lt;opc2&gt;</pre>
</div>
</div>
<div class="sect4">
<h5 id="arm-system-register-encodings"><a class="anchor" href="#arm-system-register-encodings"></a><a class="link" href="#arm-system-register-encodings">30.5.3.1. ARM system register encodings</a></h5>
<div class="paragraph">
<p>Each aarch64 system register is specified in the encoding of <a href="#arm-system-register-instructions">ARM system register instructions</a> by 5 integer numbers:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>op0</code></p>
</li>
<li>
<p><code>op1</code></p>
</li>
<li>
<p><code>CRn</code></p>
</li>
<li>
<p><code>CRm</code></p>
</li>
<li>
<p><code>op2</code></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The encodings are given on large tables in <a href="#armarm8-fa">ARMv8 architecture reference manual db</a> Chapter D12 "AArch64 System Register Encoding".</p>
</div>
<div class="paragraph">
<p>As shown in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/dump_regs.c">baremetal/arch/aarch64/dump_regs.c</a> as of LKMC 4e05b00d23c73cc4d3b83be94affdb6f28008d99, you can use the encoding parameters directly in GNU GAS assembly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>uint32_t id_isar6_el1;
__asm__ ("mrs %0, s3_0_c0_c2_7" : "=r" (id_isar6_el1) : :);
LKMC_DUMP_SYSTEM_REGS_PRINTF("ID_ISAR6_EL1 0x%" PRIX32 "\n", id_isar6_el1);</pre>
</div>
</div>
<div class="paragraph">
<p>This can be useful to refer to new system registers which your older version of GNU GAS version does not yet have a name for.</p>
</div>
<div class="paragraph">
<p>The Linux kernel also uses explicit sysreg encoding extensively since it is of course a very early user of many new system registers, this is done at <a href="https://github.com/torvalds/linux/blob/v5.4/arch/arm64/include/asm/sysreg.h"><code>arch/arm64/include/asm/sysreg.h</code> in Linux v5.4</a>.</p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-simd"><a class="anchor" href="#arm-simd"></a><a class="link" href="#arm-simd">30.6. ARM SIMD</a></h3>
<div class="paragraph">
<p>Parent section: <a href="#simd-assembly">Section 28.3, &#8220;SIMD assembly&#8221;</a></p>
</div>
<div class="sect3">
<h4 id="arm-vfp"><a class="anchor" href="#arm-vfp"></a><a class="link" href="#arm-vfp">30.6.1. ARM VFP</a></h4>
<div class="paragraph">
<p>The name for the ARMv7 and AArch32 floating point and SIMD instructions / registers.</p>
</div>
<div class="paragraph">
<p>Vector Floating Point extension.</p>
</div>
<div class="paragraph">
<p>TODO I think it was optional in ARMv7, find quote.</p>
</div>
<div class="paragraph">
<p>VFP has several revisions, named as VFPv1, VFPv2, etc. TODO: announcement dates.</p>
</div>
<div class="paragraph">
<p>As mentioned at: <a href="https://stackoverflow.com/questions/37790029/what-is-difference-between-arm64-and-armhf/48954012#48954012" class="bare">https://stackoverflow.com/questions/37790029/what-is-difference-between-arm64-and-armhf/48954012#48954012</a> the Linux kernel shows those capabilities in <code>/proc/cpuinfo</code> with flags such as <code>vfp</code>, <code>vfpv3</code> and others, see:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.18/arch/arm/kernel/setup.c#L1199" class="bare">https://github.com/torvalds/linux/blob/v4.18/arch/arm/kernel/setup.c#L1199</a></p>
</li>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.18/arch/arm64/kernel/cpuinfo.c#L95" class="bare">https://github.com/torvalds/linux/blob/v4.18/arch/arm64/kernel/cpuinfo.c#L95</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>When a certain version of VFP is present on a CPU, the compiler prefix typically contains the <code>hf</code> characters which stands for Hard Float, e.g.: <code>arm-linux-gnueabihf</code>. This means that the compiler will emit VFP instructions instead of just using software implementations.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#armarm7">ARMv7 architecture reference manual</a> Appendix D6 "Common VFP Subarchitecture Specification". It is not part of the ISA, but just an extension. TODO: that spec does not seem to have the instructions documented, and instruction like VMOV just live with the main instructions. Is VMOV part of VFP?</p>
</li>
<li>
<p><a href="https://mindplusplus.wordpress.com/2013/06/25/arm-vfp-vector-programming-part-1-introduction/" class="bare">https://mindplusplus.wordpress.com/2013/06/25/arm-vfp-vector-programming-part-1-introduction/</a></p>
</li>
<li>
<p><a href="https://en.wikipedia.org/wiki/ARM_architecture#Floating-point_(VFP" class="bare">https://en.wikipedia.org/wiki/ARM_architecture#Floating-point_(VFP</a>)</p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="arm-vfp-registers"><a class="anchor" href="#arm-vfp-registers"></a><a class="link" href="#arm-vfp-registers">30.6.1.1. ARM VFP registers</a></h5>
<div class="paragraph">
<p>TODO example</p>
</div>
<div class="paragraph">
<p><a href="#armarm8">ARMv8 architecture reference manual</a> E1.3.1 "The SIMD and floating-point register file" Figure E1-1 "SIMD and floating-point register file, AArch32 operation":</p>
</div>
<div class="literalblock">
<div class="content">
<pre>+-----+-----+-----+
| S0  |     |     |
+-----+ D0  +     |
| S1  |     |     |
+-----+-----+ Q0  |
| S2  |     |     |
+-----+ D1  +     |
| S3  |     |     |
+-----+-----+-----+
| S4  |     |     |
+-----+ D2  +     |
| S5  |     |     |
+-----+-----+ Q1  |
| S6  |     |     |
+-----+ D3  +     |
| S7  |     |     |
+-----+-----+-----+</pre>
</div>
</div>
<div class="paragraph">
<p>Note how Sn is weirdly packed inside Dn, and Dn weirdly packed inside Qn, likely for historical reasons.</p>
</div>
<div class="paragraph">
<p>And you can&#8217;t access the higher bytes at D16 or greater with Sn.</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-vadd-instruction"><a class="anchor" href="#arm-vadd-instruction"></a><a class="link" href="#arm-vadd-instruction">30.6.1.2. ARM VADD instruction</a></h5>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/vadd_scalar.S">userland/arch/arm/vadd_scalar.S</a>: see also: <a href="#floating-point-assembly">Section 28.2, &#8220;Floating point assembly&#8221;</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/vadd_vector.S">userland/arch/arm/vadd_vector.S</a>: see also: <a href="#simd-assembly">Section 28.3, &#8220;SIMD assembly&#8221;</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="arm-vcvt-instruction"><a class="anchor" href="#arm-vcvt-instruction"></a><a class="link" href="#arm-vcvt-instruction">30.6.1.3. ARM VCVT instruction</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/vcvt.S">userland/arch/arm/vcvt.S</a></p>
</div>
<div class="paragraph">
<p>Convert between integers and floating point.</p>
</div>
<div class="paragraph">
<p><a href="#armarm7">ARMv7 architecture reference manual</a> on rounding:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The floating-point to fixed-point operation uses the Round towards Zero rounding mode. The fixed-point to floating-point operation uses the Round to Nearest rounding mode.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Notice how the opcode takes two types.</p>
</div>
<div class="paragraph">
<p>E.g., in our 32-bit float to 32-bit unsigned example we use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vld1.32.f32</pre>
</div>
</div>
<div class="sect5">
<h6 id="arm-vcvtr-instruction"><a class="anchor" href="#arm-vcvtr-instruction"></a><a class="link" href="#arm-vcvtr-instruction">30.6.1.3.1. ARM VCVTR instruction</a></h6>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/vcvtr.S">userland/arch/arm/vcvtr.S</a></p>
</div>
<div class="paragraph">
<p>Like <a href="#arm-vcvt-instruction">ARM VCVT instruction</a>, but the rounding mode is selected by the FPSCR.RMode field.</p>
</div>
<div class="paragraph">
<p>Selecting rounding mode explicitly per instruction was apparently not possible in ARMv7, but was made possible in <a href="#aarch32">AArch32</a> e.g. with <a href="#armv8-aarch32-vcvta-instruction">ARMv8 AArch32 VCVTA instruction</a>.</p>
</div>
<div class="paragraph">
<p>Rounding mode selection is exposed in the ANSI C standard through <a href="https://en.cppreference.com/w/c/numeric/fenv/feround"><code>fesetround</code></a>.</p>
</div>
<div class="paragraph">
<p>TODO: is the initial rounding mode specified by the ELF standard? Could not find a reference.</p>
</div>
</div>
<div class="sect5">
<h6 id="armv8-aarch32-vcvta-instruction"><a class="anchor" href="#armv8-aarch32-vcvta-instruction"></a><a class="link" href="#armv8-aarch32-vcvta-instruction">30.6.1.3.2. ARMv8 AArch32 VCVTA instruction</a></h6>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/vcvt.S">userland/arch/arm/vcvt.S</a></p>
</div>
<div class="paragraph">
<p>Added in ARMv8 <a href="#aarch32">AArch32</a> only, not present in ARMv7.</p>
</div>
<div class="paragraph">
<p>In ARMv7, to use a non-round-to-zero rounding mode, you had to set the rounding mode with FPSCR and use the R version of the instruction e.g. <a href="#arm-vcvtr-instruction">ARM VCVTR instruction</a>.</p>
</div>
<div class="paragraph">
<p>Now in AArch32 it is possible to do it explicitly per-instruction.</p>
</div>
<div class="paragraph">
<p>Also there was no ties to away mode in ARMv7. This mode does not exist in C99 either.</p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="armv8-advanced-simd-and-floating-point-support"><a class="anchor" href="#armv8-advanced-simd-and-floating-point-support"></a><a class="link" href="#armv8-advanced-simd-and-floating-point-support">30.6.2. ARMv8 Advanced SIMD and floating-point support</a></h4>
<div class="paragraph">
<p>The <a href="#armarm8">ARMv8 architecture reference manual</a> specifies floating point and SIMD support in the main architecture at A1.5 "Advanced SIMD and floating-point support".</p>
</div>
<div class="paragraph">
<p>The feature is often refered to simply as "SIMD&amp;FP" throughout the manual.</p>
</div>
<div class="paragraph">
<p>The Linux kernel shows <code>/proc/cpuinfo</code> compatibility as <code>neon</code>, which is yet another intermediate name that came up at some point, see: <a href="#arm-neon">Section 30.6.2.2, &#8220;ARM NEON&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Vs <a href="#arm-vfp">ARM VFP</a>: <a href="https://stackoverflow.com/questions/4097034/arm-cortex-a8-whats-the-difference-between-vfp-and-neon" class="bare">https://stackoverflow.com/questions/4097034/arm-cortex-a8-whats-the-difference-between-vfp-and-neon</a></p>
</div>
<div class="sect4">
<h5 id="armv8-floating-point-availability"><a class="anchor" href="#armv8-floating-point-availability"></a><a class="link" href="#armv8-floating-point-availability">30.6.2.1. ARMv8 floating point availability</a></h5>
<div class="paragraph">
<p>Support is semi-mandatory. <a href="#armarm8">ARMv8 architecture reference manual</a> A1.5 "Advanced SIMD and floating-point support":</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>ARMv8 can support the following levels of support for Advanced SIMD and floating-point instructions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Full SIMD and floating-point support without exception trapping.</p>
</li>
<li>
<p>Full SIMD and floating-point support with exception trapping.</p>
</li>
<li>
<p>No floating-point or SIMD support. This option is licensed only for implementations targeting specialized markets.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Note: All systems that support standard operating systems with rich application environments provide hardware
support for Advanced SIMD and floating-point. It is a requirement of the ARM Procedure Call Standard for
AArch64, see Procedure Call Standard for the ARM 64-bit Architecture.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Therefore it is in theory optional, but highly available.</p>
</div>
<div class="paragraph">
<p>This is unlike ARMv7, where floating point is completely optional through <a href="#arm-vfp">ARM VFP</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-neon"><a class="anchor" href="#arm-neon"></a><a class="link" href="#arm-neon">30.6.2.2. ARM NEON</a></h5>
<div class="paragraph">
<p>Just an informal name for the "Advanced SIMD instructions"? Very confusing.</p>
</div>
<div class="paragraph">
<p><a href="#armarm8">ARMv8 architecture reference manual</a> F2.9 "Additional information about Advanced SIMD and floating-point instructions" says:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The Advanced SIMD architecture, its associated implementations, and supporting software, are commonly referred to as NEON technology.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p><a href="https://developer.arm.com/technologies/neon" class="bare">https://developer.arm.com/technologies/neon</a> mentions that is is present on both ARMv7 and ARMv8:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>NEON technology was introduced to the Armv7-A and Armv7-R profiles. It is also now an extension to the Armv8-A and Armv8-R profiles.</p>
</div>
</blockquote>
</div>
</div>
</div>
<div class="sect3">
<h4 id="armv8-aarch64-floating-point-registers"><a class="anchor" href="#armv8-aarch64-floating-point-registers"></a><a class="link" href="#armv8-aarch64-floating-point-registers">30.6.3. ARMv8 AArch64 floating point registers</a></h4>
<div class="paragraph">
<p>TODO example.</p>
</div>
<div class="paragraph">
<p><a href="#armarm8">ARMv8 architecture reference manual</a> B1.2.1 "Registers in AArch64 state" describes the registers:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>32 SIMD&amp;FP registers, V0 to V31. Each register can be accessed as:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>A 128-bit register named Q0 to Q31.</p>
</li>
<li>
<p>A 64-bit register named D0 to D31.</p>
</li>
<li>
<p>A 32-bit register named S0 to S31.</p>
</li>
<li>
<p>A 16-bit register named H0 to H31.</p>
</li>
<li>
<p>An 8-bit register named B0 to B31.</p>
</li>
</ul>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Notice how Sn is very different between v7 <a href="#arm-vfp-registers">ARM VFP registers</a> and v8! In v7 it goes across Dn, and in v8 inside each Dn:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>128                         64                  32      16  8   0
+---------------------------+-------------------+-------+---+---+
|                           Vn                                  |
+---------------------------------------------------------------+
|                           Qn                                  |
+---------------------------+-----------------------------------+
                            |                   Dn              |
                            +-----------------------------------+
                                                |       Sn      |
                                                +---------------+
                                                        |   Hn  |
                                                        +-------+
                                                            |Bn |
                                                            +---+</pre>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-add-vector-instruction"><a class="anchor" href="#armv8-aarch64-add-vector-instruction"></a><a class="link" href="#armv8-aarch64-add-vector-instruction">30.6.3.1. ARMv8 aarch64 add vector instruction</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/add_vector.S">userland/arch/aarch64/add_vector.S</a></p>
</div>
<div class="paragraph">
<p>Good first instruction to learn SIMD: <a href="#simd-assembly">SIMD assembly</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-fadd-instruction"><a class="anchor" href="#armv8-aarch64-fadd-instruction"></a><a class="link" href="#armv8-aarch64-fadd-instruction">30.6.3.2. ARMv8 aarch64 FADD instruction</a></h5>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/fadd_vector.S">userland/arch/aarch64/fadd_vector.S</a>: see also: <a href="#simd-assembly">Section 28.3, &#8220;SIMD assembly&#8221;</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/fadd_scalar.S">userland/arch/aarch64/fadd_scalar.S</a>: see also: <a href="#floating-point-assembly">Section 28.2, &#8220;Floating point assembly&#8221;</a></p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="arm-fadd-vs-vadd"><a class="anchor" href="#arm-fadd-vs-vadd"></a><a class="link" href="#arm-fadd-vs-vadd">30.6.3.2.1. ARM FADD vs VADD</a></h6>
<div class="paragraph">
<p>It is very confusing, but FADDS and FADDD in Aarch32 are <a href="#gnu-gas-assembler-arm-unified-syntax">pre-UAL</a> for <code>vadd.f32</code> and <code>vadd.f64</code> which we use in this tutorial, see: <a href="#arm-vadd-instruction">Section 30.6.1.2, &#8220;ARM VADD instruction&#8221;</a></p>
</div>
<div class="paragraph">
<p>The same goes for most ARMv7 mnemonics: <code>f*</code> is old, and <code>v*</code> is the newer better syntax.</p>
</div>
<div class="paragraph">
<p>But then, in ARMv8, they decided to use <a href="#armv8-aarch64-fadd-instruction">ARMv8 aarch64 FADD instruction</a> as the main floating point add name, and get rid of VADD!</p>
</div>
<div class="paragraph">
<p>Also keep in mind that fused multiply add is FMADD.</p>
</div>
<div class="paragraph">
<p>Examples at: <a href="#simd-assembly">Section 28.3, &#8220;SIMD assembly&#8221;</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="armv8-aarch64-ld2-instruction"><a class="anchor" href="#armv8-aarch64-ld2-instruction"></a><a class="link" href="#armv8-aarch64-ld2-instruction">30.6.3.3. ARMv8 aarch64 LD2 instruction</a></h5>
<div class="paragraph">
<p>Example: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/ld2.S">userland/arch/aarch64/ld2.S</a></p>
</div>
<div class="paragraph">
<p>We can load multiple vectors interleaved from memory in one single instruction!</p>
</div>
<div class="paragraph">
<p>This is why the <code>ldN</code> instructions take an argument list denoted by <code>{}</code> for the registers, much like armv7 <a href="#arm-ldmia-instruction">ARM LDMIA instruction</a>.</p>
</div>
<div class="paragraph">
<p>There are analogous LD3 and LD4 instruction.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-simd-bibliography"><a class="anchor" href="#arm-simd-bibliography"></a><a class="link" href="#arm-simd-bibliography">30.6.4. ARM SIMD bibliography</a></h4>
<div class="ulist">
<ul>
<li>
<p>GNU GAS tests under <a href="https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=tree;f=gas/testsuite/gas/aarch64;hb=00f223631fa9803b783515a2f667f86997e2cdbe"><code>gas/testsuite/gas/aarch64</code></a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/2851421/is-there-a-good-reference-for-arm-neon-intrinsics" class="bare">https://stackoverflow.com/questions/2851421/is-there-a-good-reference-for-arm-neon-intrinsics</a></p>
</li>
<li>
<p>assembly optimized libraries:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/projectNe10/Ne10" class="bare">https://github.com/projectNe10/Ne10</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="arm-sve"><a class="anchor" href="#arm-sve"></a><a class="link" href="#arm-sve">30.6.5. ARM SVE</a></h4>
<div class="paragraph">
<p>Scalable Vector Extension.</p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/sve.S">userland/arch/aarch64/sve.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>To understand it, the first thing you have to look at is the execution example at Fig 1 of: <a href="https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf" class="bare">https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf</a></p>
</div>
<div class="paragraph">
<p>aarch64 only, newer than <a href="#arm-neon">ARM NEON</a>.</p>
</div>
<div class="paragraph">
<p>It is called Scalable because it does not specify the vector width! Therefore we don&#8217;t have to worry about new vector width instructions every few years! Hurray!</p>
</div>
<div class="paragraph">
<p>The instructions then allow:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>incrementing loop index by the vector length without explicitly hardcoding it</p>
</li>
<li>
<p>when the last loop is reached, extra bytes that are not multiples of the vector length get automatically masked out by the predicate register, and have no effect</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Added to QEMU in 3.0.0 and gem5 in 2019 Q3.</p>
</div>
<div class="paragraph">
<p>TODO announcement date. Possibly 2017: <a href="https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf" class="bare">https://alastairreid.github.io/papers/sve-ieee-micro-2017.pdf</a> There is also a 2016 mention: <a href="https://community.arm.com/tools/hpc/b/hpc/posts/technology-update-the-scalable-vector-extension-sve-for-the-armv8-a-architecture" class="bare">https://community.arm.com/tools/hpc/b/hpc/posts/technology-update-the-scalable-vector-extension-sve-for-the-armv8-a-architecture</a></p>
</div>
<div class="paragraph">
<p>The Linux kernel shows <code>/proc/cpuinfo</code> compatibility as <code>sve</code>.</p>
</div>
<div class="paragraph">
<p>Official spec: <a href="https://developer.arm.com/docs/100891/latest/sve-overview/introducing-sve" class="bare">https://developer.arm.com/docs/100891/latest/sve-overview/introducing-sve</a></p>
</div>
<div class="paragraph">
<p>SVE support is indicated by <code>ID_AA64PFR0_EL1.SVE</code> which is dumped from <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/dump_regs.c">baremetal/arch/aarch64/dump_regs.c</a>.</p>
</div>
<div class="paragraph">
<p>Using SVE normally requires setting the CPACR_EL1.FPEN and ZEN bits, which as as of lkmc 29fd625f3fda79f5e0ee6cac43517ba74340d513 + 1 we also enable in our <a href="#baremetal-bootloaders">Baremetal bootloaders</a>, see also: <a href="#aarch64-baremetal-neon-setup">aarch64 baremetal NEON setup</a>.</p>
</div>
<div class="sect4">
<h5 id="arm-sve-vaddl-instruction"><a class="anchor" href="#arm-sve-vaddl-instruction"></a><a class="link" href="#arm-sve-vaddl-instruction">30.6.5.1. ARM SVE VADDL instruction</a></h5>
<div class="paragraph">
<p>Get the SVE vector length. The following programs do that and print it to stdout:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/sve_addvl.c">userland/arch/aarch64/inline_asm/sve_addvl.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/sve_addvl.S">userland/arch/aarch64/sve_addvl.S</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="change-arm-sve-vector-length-in-emulators"><a class="anchor" href="#change-arm-sve-vector-length-in-emulators"></a><a class="link" href="#change-arm-sve-vector-length-in-emulators">30.6.5.2. Change ARM SVE vector length in emulators</a></h5>
<div class="paragraph">
<p>gem5 covered at: <a href="https://stackoverflow.com/questions/57692765/how-to-change-the-gem5-arm-sve-vector-length" class="bare">https://stackoverflow.com/questions/57692765/how-to-change-the-gem5-arm-sve-vector-length</a></p>
</div>
<div class="paragraph">
<p>It is fun to observe this directly with the <a href="#arm-sve-vaddl-instruction">ARM SVE VADDL instruction</a> in SE:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --userland userland/arch/aarch64/sve_addvl.S --emulator gem5 -- --param 'system.cpu[:].isa[:].sve_vl_se = 1'
./run --arch aarch64 --userland userland/arch/aarch64/sve_addvl.S --emulator gem5 -- --param 'system.cpu[:].isa[:].sve_vl_se = 2'
./run --arch aarch64 --userland userland/arch/aarch64/sve_addvl.S --emulator gem5 -- --param 'system.cpu[:].isa[:].sve_vl_se = 4'</pre>
</div>
</div>
<div class="paragraph">
<p>which consecutively:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x0000000000000080
0x0000000000000100
0x0000000000000200</pre>
</div>
</div>
<div class="paragraph">
<p>which are multiples of 128.</p>
</div>
<div class="paragraph">
<p>TODO how to set it on QEMU at runtime? As of LKMC 37b93ecfbb5a1fcbd0c631dd0b42c5b9f2f8a89a + 1 QEMU outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0x0000000000000800</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="sve-bibliography"><a class="anchor" href="#sve-bibliography"></a><a class="link" href="#sve-bibliography">30.6.5.3. SVE bibliography</a></h5>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.rico.cat/files/ICS18-gem5-sve-tutorial.pdf" class="bare">https://www.rico.cat/files/ICS18-gem5-sve-tutorial.pdf</a> step by step of a complete code execution examples, the best initial tutorial so far</p>
</li>
<li>
<p><a href="https://static.docs.arm.com/dui0965/c/DUI0965C_scalable_vector_extension_guide.pdf" class="bare">https://static.docs.arm.com/dui0965/c/DUI0965C_scalable_vector_extension_guide.pdf</a></p>
</li>
<li>
<p><a href="https://developer.arm.com/products/software-development-tools/hpc/documentation/writing-inline-sve-assembly" class="bare">https://developer.arm.com/products/software-development-tools/hpc/documentation/writing-inline-sve-assembly</a> quick inlining guide</p>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="sve-spec"><a class="anchor" href="#sve-spec"></a><a class="link" href="#sve-spec">30.6.5.3.1. SVE spec</a></h6>
<div class="paragraph">
<p><a href="#armarm8">ARMv8 architecture reference manual</a> A1.7 "ARMv8 architecture extensions" says:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>SVE is an optional extension to ARMv8.2. That is, SVE requires the implementation of ARMv8.2.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>A1.7.8 "The Scalable Vector Extension (SVE)": then says that only changes to the existing registers are described in that manual, and that you should look instead at the "ARM Architecture Reference Manual Supplement, The Scalable Vector Extension (SVE), for ARMv8-A."</p>
</div>
<div class="paragraph">
<p>We then download the zip from: <a href="https://developer.arm.com/docs/ddi0584/latest/arm-architecture-reference-manual-supplement-the-scalable-vector-extension-sve-for-armv8-a" class="bare">https://developer.arm.com/docs/ddi0584/latest/arm-architecture-reference-manual-supplement-the-scalable-vector-extension-sve-for-armv8-a</a> and it contains the PDF: <code>DDI0584A_d_SVE_supp_armv8A.pdf</code> which we use here.</p>
</div>
<div class="paragraph">
<p>That document then describes the SVE instructions and registers.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-thread-synchronization-primitives"><a class="anchor" href="#arm-thread-synchronization-primitives"></a><a class="link" href="#arm-thread-synchronization-primitives">30.7. ARM thread synchronization primitives</a></h3>
<div class="paragraph">
<p>Parent section: <a href="#userland-multithreading">Userland multithreading</a>.</p>
</div>
<div class="sect3">
<h4 id="arm-ldxr-and-stxr-instructions"><a class="anchor" href="#arm-ldxr-and-stxr-instructions"></a><a class="link" href="#arm-ldxr-and-stxr-instructions">30.7.1. ARM LDXR and STXR instructions</a></h4>
<div class="paragraph">
<p>Parent section: <a href="#atomic-cpp">atomic.cpp</a></p>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldaxr_stlxr.cpp">userland/cpp/atomic/aarch64_ldaxr_stlxr.cpp</a></p>
</div>
<div class="paragraph">
<p>LDXR and STXR vs LDAXR and STLXR: <a href="https://stackoverflow.com/questions/21535058/arm64-ldxr-stxr-vs-ldaxr-stlxr" class="bare">https://stackoverflow.com/questions/21535058/arm64-ldxr-stxr-vs-ldaxr-stlxr</a> TODO understand better and example.</p>
</div>
<div class="paragraph">
<p>LDXR and STXR for a so-called "Load-link/store-conditional" (LLSC) pattern: <a href="https://en.wikipedia.org/wiki/Load-link/store-conditional" class="bare">https://en.wikipedia.org/wiki/Load-link/store-conditional</a> which appears in many RISC ISAs.</p>
</div>
<div class="paragraph">
<p>This pattern makes it such that basically:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>LDXR marks an address for exclusive access by the current CPU</p>
</li>
<li>
<p>STXR:</p>
<div class="ulist">
<ul>
<li>
<p>marks the address as not being exclusive to other CPUs that may have done LDXR before</p>
</li>
<li>
<p>loads fine if the address is still marked as exclusive, and stores 0 on a third register for success</p>
</li>
<li>
<p>fails to load if the address is not, and stores 1 on the third register for failure</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>In case of failure, we just have to loop back to just before the LDXR and try again.</p>
</div>
<div class="paragraph">
<p>This is therefore basically a spinlock and should only be used to cover very short critical sections such as atomic increments.</p>
</div>
<div class="paragraph">
<p>C++ <code>std::atomic</code> uses this for increments before v8.1 <a href="#arm-lse">ARM Large System Extensions (LSE)</a>: <a href="https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291" class="bare">https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291</a></p>
</div>
</div>
<div class="sect3">
<h4 id="arm-lse"><a class="anchor" href="#arm-lse"></a><a class="link" href="#arm-lse">30.7.2. ARM Large System Extensions (LSE)</a></h4>
<div class="paragraph">
<p>Set of atomic and synchronization primitives added in <a href="#armv8-1-architecture-extension">ARMv8.1 architecture extension</a>.</p>
</div>
<div class="paragraph">
<p>Documented at <a href="#armarm8-db">ARMv8 architecture reference manual db</a> "ARMv8.1-LSE, ARMv8.1 Large System Extensions"</p>
</div>
<div class="ulist">
<ul>
<li>
<p>LDADD: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/aarch64_ldadd.cpp">userland/cpp/atomic/aarch64_ldadd.cpp</a>, see also: <a href="#atomic-cpp">atomic.cpp</a>. Kernel inspiration: <a href="https://github.com/torvalds/linux/blob/v5.4/arch/arm64/include/asm/atomic_lse.h#L56" class="bare">https://github.com/torvalds/linux/blob/v5.4/arch/arm64/include/asm/atomic_lse.h#L56</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/" class="bare">https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="armv8-architecture-extensions"><a class="anchor" href="#armv8-architecture-extensions"></a><a class="link" href="#armv8-architecture-extensions">30.8. ARMv8 architecture extensions</a></h3>
<div class="sect3">
<h4 id="armv8-1-architecture-extension"><a class="anchor" href="#armv8-1-architecture-extension"></a><a class="link" href="#armv8-1-architecture-extension">30.8.1. ARMv8.1 architecture extension</a></h4>
<div class="paragraph">
<p><a href="#armarm8-db">ARMv8 architecture reference manual db</a> A1.7.3 "The ARMv8.1 architecture extension"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-lse">ARM Large System Extensions (LSE)</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-pmu"><a class="anchor" href="#arm-pmu"></a><a class="link" href="#arm-pmu">30.9. ARM PMU</a></h3>
<div class="paragraph">
<p>The PMU (Performance Monitor Unit) is an unit in the ARM CPU that counts performance events of interest. These can be used to benchmark, and sometimes debug, code running on ARM CPUs.</p>
</div>
<div class="paragraph">
<p>It is documented at <a href="#armarm8-fa">ARMv8 architecture reference manual db</a> Chapter D7 "The Performance Monitors Extension"&gt;</p>
</div>
<div class="paragraph">
<p>The <a href="#linux-kernel">Linux kernel</a> exposes some (all?) of those events through the arch-agnostic <a href="#perf-event-open"><code>perf_event_open</code> system call</a> system call.</p>
</div>
<div class="paragraph">
<p>Exposing the PMU to Linux v5.9.2 requires a <a href="#dtb-files">DTB</a> entry of type:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>pmu {
        compatible = "arm,armv8-pmuv3";
        interrupts = &lt;0x01 0x04 0xf04&gt;;
};</pre>
</div>
</div>
<div class="paragraph">
<p>and if sucessful, a boot message shows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>&lt;6&gt;[    0.044391] hw perfevents: enabled with armv8_pmuv3 PMU driver, 32 counters available</pre>
</div>
</div>
<div class="paragraph">
<p>The PMU is exposed through <a href="#arm-system-register-instructions">ARM system register instructions</a>, with registers that start with the prefix <code>PM*</code>.</p>
</div>
<div class="paragraph">
<p>&lt;6&gt;[    0.044391] hw perfevents: enabled with armv8_pmuv3 PMU driver, 32 counters available</p>
</div>
<div class="paragraph">
<p><a href="#armarm8-fa">ARMv8 architecture reference manual db</a> D7.11.3 "Common event numbers" gives the available standardized events. Address space is also reverved for vendor extensions. For example, from it we see that the instruction count is documented at:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>0x0008, INST_RETIRED, Instruction architecturally executed</p>
</div>
<div class="paragraph">
<p>The counter increments for every architecturally executed instruction.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>where "architecturally executed" is a reference to the possibility of <a href="#out-of-order-execution">Out-of-order execution</a> in the implementation, which leads to some instructions being executed speculatively, but not have any side effects in the end.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/using-the-arm-performance-monitor-unit-pmu-linux-driver" class="bare">https://community.arm.com/developer/ip-products/system/b/embedded-blog/posts/using-the-arm-performance-monitor-unit-pmu-linux-driver</a></p>
</div>
<div class="sect3">
<h4 id="arm-pmccntr-register"><a class="anchor" href="#arm-pmccntr-register"></a><a class="link" href="#arm-pmccntr-register">30.9.1. ARM PMCCNTR register</a></h4>
<div class="paragraph">
<p>TODO We didn&#8217;t manage to find a working ARM analogue to <a href="#x86-rdtsc-instruction">x86 RDTSC instruction</a>: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/kernel_modules/pmccntr.c">kernel_modules/pmccntr.c</a> is oopsing, and even it if weren&#8217;t, it likely won&#8217;t give the cycle count since boot since it needs to be activate before it starts counting anything:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/40454157/is-there-an-equivalent-instruction-to-rdtsc-in-arm" class="bare">https://stackoverflow.com/questions/40454157/is-there-an-equivalent-instruction-to-rdtsc-in-arm</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/31620375/arm-cortex-a7-returning-pmccntr-0-in-kernel-mode-and-illegal-instruction-in-u/31649809#31649809" class="bare">https://stackoverflow.com/questions/31620375/arm-cortex-a7-returning-pmccntr-0-in-kernel-mode-and-illegal-instruction-in-u/31649809#31649809</a></p>
</li>
<li>
<p><a href="https://blog.regehr.org/archives/794" class="bare">https://blog.regehr.org/archives/794</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-assembly-bibliography"><a class="anchor" href="#arm-assembly-bibliography"></a><a class="link" href="#arm-assembly-bibliography">30.10. ARM assembly bibliography</a></h3>
<div class="sect3">
<h4 id="arm-non-official-bibliography"><a class="anchor" href="#arm-non-official-bibliography"></a><a class="link" href="#arm-non-official-bibliography">30.10.1. ARM non-official bibliography</a></h4>
<div class="paragraph">
<p>Good getting started tutorials:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://www.davespace.co.uk/arm/introduction-to-arm/" class="bare">http://www.davespace.co.uk/arm/introduction-to-arm/</a></p>
</li>
<li>
<p><a href="https://azeria-labs.com/writing-arm-assembly-part-1/" class="bare">https://azeria-labs.com/writing-arm-assembly-part-1/</a></p>
</li>
<li>
<p><a href="https://thinkingeek.com/arm-assembler-raspberry-pi/" class="bare">https://thinkingeek.com/arm-assembler-raspberry-pi/</a></p>
</li>
<li>
<p><a href="http://bob.cs.sonoma.edu/IntroCompOrg-RPi/app-make.html" class="bare">http://bob.cs.sonoma.edu/IntroCompOrg-RPi/app-make.html</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="arm-official-bibliography"><a class="anchor" href="#arm-official-bibliography"></a><a class="link" href="#arm-official-bibliography">30.10.2. ARM official bibliography</a></h4>
<div class="paragraph">
<p>The official manuals were stored in <a href="http://infocenter.arm.com" class="bare">http://infocenter.arm.com</a> but as of 2017 they started to slowly move to <a href="https://developer.arm.com" class="bare">https://developer.arm.com</a>.</p>
</div>
<div class="paragraph">
<p>Each revision of a document has a "ARM DDI" unique document identifier.</p>
</div>
<div class="paragraph">
<p>The "ARM Architecture Reference Manuals" are the official canonical ISA documentation document. In this repository, we always reference the following revisions:</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://www.quora.com/Where-can-I-find-the-official-documentation-of-ARM-instruction-set-architectures-ISAs" class="bare">https://www.quora.com/Where-can-I-find-the-official-documentation-of-ARM-instruction-set-architectures-ISAs</a></p>
</div>
<div class="sect4">
<h5 id="armarm7"><a class="anchor" href="#armarm7"></a><a class="link" href="#armarm7">30.10.2.1. ARMv7 architecture reference manual</a></h5>
<div class="paragraph">
<p><a href="https://developer.arm.com/products/architecture/a-profile/docs/ddi0406/latest/arm-architecture-reference-manual-armv7-a-and-armv7-r-edition" class="bare">https://developer.arm.com/products/architecture/a-profile/docs/ddi0406/latest/arm-architecture-reference-manual-armv7-a-and-armv7-r-edition</a></p>
</div>
<div class="paragraph">
<p>The official comprehensive ARMv7 reference.</p>
</div>
<div class="paragraph">
<p>We use by default: DDI 0406C.d: <a href="https://static.docs.arm.com/ddi0406/cd/DDI0406C_d_armv7ar_arm.pdf" class="bare">https://static.docs.arm.com/ddi0406/cd/DDI0406C_d_armv7ar_arm.pdf</a></p>
</div>
</div>
<div class="sect4">
<h5 id="armarm8"><a class="anchor" href="#armarm8"></a><a class="link" href="#armarm8">30.10.2.2. ARMv8 architecture reference manual</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf" class="bare">https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf</a></p>
</div>
<div class="paragraph">
<p>Latest version: <a href="https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile" class="bare">https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile</a></p>
</div>
<div class="paragraph">
<p>Versions are determined by two letteres in lexicographical order, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a</p>
</li>
<li>
<p>af</p>
</li>
<li>
<p>aj</p>
</li>
<li>
<p>aj</p>
</li>
<li>
<p>b</p>
</li>
<li>
<p>ba</p>
</li>
<li>
<p>bb</p>
</li>
<li>
<p>ca</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The link: <a href="https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf" class="bare">https://static.docs.arm.com/ddi0487/ca/DDI0487C_a_armv8_arm.pdf</a> is the <code>ca</code> version for example.</p>
</div>
<div class="paragraph">
<p>The official comprehensive ARMv8 reference.</p>
</div>
<div class="paragraph">
<p>ISA quick references can be found in some places:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://web.archive.org/web/20161009122630/http://infocenter.arm.com/help/topic/com.arm.doc.qrc0001m/QRC0001_UAL.pdf" class="bare">https://web.archive.org/web/20161009122630/http://infocenter.arm.com/help/topic/com.arm.doc.qrc0001m/QRC0001_UAL.pdf</a></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="armarm8-db"><a class="anchor" href="#armarm8-db"></a><a class="link" href="#armarm8-db">30.10.2.3. ARMv8 architecture reference manual db</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/ddi0487/db/DDI0487D_b_armv8_arm.pdf" class="bare">https://static.docs.arm.com/ddi0487/db/DDI0487D_b_armv8_arm.pdf</a></p>
</div>
</div>
<div class="sect4">
<h5 id="armarm8-fa"><a class="anchor" href="#armarm8-fa"></a><a class="link" href="#armarm8-fa">30.10.2.4. ARMv8 architecture reference manual db</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/ddi0487/fa/DDI0487F_a_armv8_arm.pdf" class="bare">https://static.docs.arm.com/ddi0487/fa/DDI0487F_a_armv8_arm.pdf</a></p>
</div>
</div>
<div class="sect4">
<h5 id="armv8-programmers-guide"><a class="anchor" href="#armv8-programmers-guide"></a><a class="link" href="#armv8-programmers-guide">30.10.2.5. Programmer&#8217;s Guide for ARMv8-A</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/den0024/a/DEN0024A_v8_architecture_PG.pdf" class="bare">https://static.docs.arm.com/den0024/a/DEN0024A_v8_architecture_PG.pdf</a></p>
</div>
<div class="paragraph">
<p>A more terse human readable introduction to the ARM architecture than the reference manuals.</p>
</div>
<div class="paragraph">
<p>Does not have as many assembly code examples as you&#8217;d hope however&#8230;&#8203;</p>
</div>
<div class="paragraph">
<p>Latest version at: <a href="https://developer.arm.com/docs/den0024/latest/preface" class="bare">https://developer.arm.com/docs/den0024/latest/preface</a></p>
</div>
</div>
<div class="sect4">
<h5 id="arm-a64-instruction-set-architecture-future-architecture-technologies-in-the-a-architecture-profile-documentation"><a class="anchor" href="#arm-a64-instruction-set-architecture-future-architecture-technologies-in-the-a-architecture-profile-documentation"></a><a class="link" href="#arm-a64-instruction-set-architecture-future-architecture-technologies-in-the-a-architecture-profile-documentation">30.10.2.6. Arm A64 Instruction Set Architecture: Future Architecture Technologies in the A architecture profile Documentation</a></h5>
<div class="paragraph">
<p><a href="https://developer.arm.com/docs/ddi0602/b" class="bare">https://developer.arm.com/docs/ddi0602/b</a></p>
</div>
<div class="paragraph">
<p>This page contains the documentation of architecture features that were publicly announced but haven&#8217;t been merged into the main spec yet.</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-processor-documentation"><a class="anchor" href="#arm-processor-documentation"></a><a class="link" href="#arm-processor-documentation">30.10.2.7. ARM processor documentation</a></h5>
<div class="paragraph">
<p>ARM also releases documentation specific to each given processor.</p>
</div>
<div class="paragraph">
<p>This adds extra details to the more portable <a href="#armarm8">ARMv8 architecture reference manual</a> ISA documentation.</p>
</div>
<div class="paragraph">
<p>For every processor, there are basically two key documents:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>technical reference manual, e.g.: <a href="#arm-cortex-a77-trm">Arm Cortex‑A77 Technical Reference Manual r1p1</a></p>
</li>
<li>
<p>software optimization guide, e.g.: <a href="#arm-cortex-a77-sog">Arm Cortex‑A77 Software Optimization Guide r1p1</a></p>
<div class="paragraph">
<p>This contains some approximate instruction latencies and pipeline properties.</p>
</div>
</li>
</ul>
</div>
<div class="sect5">
<h6 id="arm-cortex15-trm"><a class="anchor" href="#arm-cortex15-trm"></a><a class="link" href="#arm-cortex15-trm">30.10.2.7.1. ARM Cortex-A15 MPCore Processor Technical Reference Manual r4p0</a></h6>
<div class="paragraph">
<p><a href="http://infocenter.arm.com/help/topic/com.arm.doc.ddi0438i/DDI0438I_cortex_a15_r4p0_trm.pdf" class="bare">http://infocenter.arm.com/help/topic/com.arm.doc.ddi0438i/DDI0438I_cortex_a15_r4p0_trm.pdf</a></p>
</div>
<div class="paragraph">
<p>2013.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="arm-cortex-a77-trm"><a class="anchor" href="#arm-cortex-a77-trm"></a><a class="link" href="#arm-cortex-a77-trm">30.10.2.8. Arm Cortex‑A77 Technical Reference Manual r1p1</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/101111/0101/arm_cortex_a77_trm_101111_0101_04_en.pdf" class="bare">https://static.docs.arm.com/101111/0101/arm_cortex_a77_trm_101111_0101_04_en.pdf</a></p>
</div>
</div>
<div class="sect4">
<h5 id="arm-cortex-a77-sog"><a class="anchor" href="#arm-cortex-a77-sog"></a><a class="link" href="#arm-cortex-a77-sog">30.10.2.9. Arm Cortex‑A77 Software Optimization Guide r1p1</a></h5>
<div class="paragraph">
<p><a href="https://static.docs.arm.com/swog011050/c/Arm_Cortex-A77_Software_Optimization_Guide.pdf" class="bare">https://static.docs.arm.com/swog011050/c/Arm_Cortex-A77_Software_Optimization_Guide.pdf</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="elf"><a class="anchor" href="#elf"></a><a class="link" href="#elf">31. ELF</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format" class="bare">https://en.wikipedia.org/wiki/Executable_and_Linkable_Format</a></p>
</div>
<div class="paragraph">
<p>This is the main format for executables, object files (<code>.o</code>) and shared libraries (<code>.so</code>) in Linux.</p>
</div>
<div class="paragraph">
<p>An introduction to the format can be found at: <a href="https://cirosantilli.com/elf-hello-world" class="bare">https://cirosantilli.com/elf-hello-world</a></p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="ieee-754"><a class="anchor" href="#ieee-754"></a><a class="link" href="#ieee-754">32. IEEE 754</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/IEEE_754" class="bare">https://en.wikipedia.org/wiki/IEEE_754</a></p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/x86_64/ieee754.S">userland/arch/x86_64/ieee754.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/float.h">lkmc/float.h</a>. Bibliography: <a href="https://stackoverflow.com/questions/52905648/how-to-use-hexadecimal-floating-point-literals-in-gnu-gas/56818851#56818851" class="bare">https://stackoverflow.com/questions/52905648/how-to-use-hexadecimal-floating-point-literals-in-gnu-gas/56818851#56818851</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/8341395/what-is-a-subnormal-floating-point-number/53203428#53203428" class="bare">https://stackoverflow.com/questions/8341395/what-is-a-subnormal-floating-point-number/53203428#53203428</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/18118408/what-is-difference-between-quiet-nan-and-signaling-nan/55648118#55648118" class="bare">https://stackoverflow.com/questions/18118408/what-is-difference-between-quiet-nan-and-signaling-nan/55648118#55648118</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/2618059/in-java-what-does-nan-mean/55673220#55673220" class="bare">https://stackoverflow.com/questions/2618059/in-java-what-does-nan-mean/55673220#55673220</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="baremetal"><a class="anchor" href="#baremetal"></a><a class="link" href="#baremetal">33. Baremetal</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Getting started at: <a href="#baremetal-setup">Section 2.9, &#8220;Baremetal setup&#8221;</a></p>
</div>
<div class="sect2">
<h3 id="baremetal-gdb-step-debug"><a class="anchor" href="#baremetal-gdb-step-debug"></a><a class="link" href="#baremetal-gdb-step-debug">33.1. Baremetal GDB step debug</a></h3>
<div class="paragraph">
<p>GDB step debug works on baremetal exactly as it does on the Linux kernel, which is described at: <a href="#gdb">Section 3, &#8220;GDB step debug&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Except that is is even cooler here since we can easily control and understand every single instruction that is being run!</p>
</div>
<div class="paragraph">
<p>For example, on the first shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/c/hello.c --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>then on the second shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm --baremetal userland/c/hello.c -- main</pre>
</div>
</div>
<div class="paragraph">
<p>Or if you are a <a href="#tmux">tmux pro</a>, do everything in one go with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/c/hello.c --gdb</pre>
</div>
</div>
<div class="paragraph">
<p>Alternatively, to start from the very first executed instruction of our tiny <a href="#baremetal-bootloaders">Baremetal bootloaders</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch arm \
  --baremetal userland/c/hello.c \
  --gdb-wait \
  --tmux-args=--no-continue \
;</pre>
</div>
</div>
<div class="paragraph">
<p>analogously to what is done for <a href="#freestanding-programs">Freestanding programs</a>.</p>
</div>
<div class="paragraph">
<p>Now you can just <code>stepi</code> to when jumping into main to go to the C code in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a>.</p>
</div>
<div class="paragraph">
<p>This is specially interesting for the executables that don&#8217;t use the bootloader from under <code>baremetal/arch/&lt;arch&gt;/no_bootloader/*.S</code>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch arm \
  --baremetal baremetal/arch/arm/no_bootloader/semihost_exit.S \
  --gdb-wait \
  --tmux-args=--no-continue \
;</pre>
</div>
</div>
<div class="paragraph">
<p>The cool thing about those examples is that you start at the very first instruction of your program, which gives more control.</p>
</div>
<div class="paragraph">
<p>Examples without bootloader are somewhat analogous to user mode <a href="#freestanding-programs">Freestanding programs</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-bootloaders"><a class="anchor" href="#baremetal-bootloaders"></a><a class="link" href="#baremetal-bootloaders">33.2. Baremetal bootloaders</a></h3>
<div class="paragraph">
<p>As can be seen from <a href="#baremetal-gdb-step-debug">Baremetal GDB step debug</a>, all examples under <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/">baremetal/</a>, with the exception of <code>baremetal/arch/&lt;arch&gt;/no_bootloader</code>, start from our tiny bootloaders:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/arm.S">baremetal/lib/arm.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/aarch64.S">baremetal/lib/aarch64.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Out simplistic bootloaders basically setup up just enough system state to allow calling:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>C functions such as <code>exit</code> from the assembly examples</p>
</li>
<li>
<p>the <code>main</code> of C examples itself</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The most important things that we setup in the bootloaders are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the stack pointer</p>
</li>
<li>
<p>NEON: <a href="#aarch64-baremetal-neon-setup">Section 33.11.2, &#8220;aarch64 baremetal NEON setup&#8221;</a></p>
</li>
<li>
<p>TODO: we don&#8217;t do this currently but maybe we should setup BSS</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The C functions that become available as a result are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Newlib functions implemented at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/syscalls.c">baremetal/lib/syscalls.c</a></p>
</li>
<li>
<p><code>lkmc_</code> non-Newlib functions implemented at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc.c">lkmc.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It is not possible to call those C functions from the examples that don&#8217;t use a bootloader.</p>
</div>
<div class="paragraph">
<p>For this reason, we tend to create examples with bootloaders, as it is easier to write them portably.</p>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-linker-script"><a class="anchor" href="#baremetal-linker-script"></a><a class="link" href="#baremetal-linker-script">33.3. Baremetal linker script</a></h3>
<div class="paragraph">
<p>For things to work in baremetal, we often have to layout memory in specific ways.</p>
</div>
<div class="paragraph">
<p>Notably, since we start with <a href="#arm-paging">paging</a> disabled, there are more constraints on where memory can or cannot go.</p>
</div>
<div class="paragraph">
<p>Especially for C programs, this memory layout is specified by a "linker script", which is present at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/link.ld">baremetal/link.ld</a></p>
</div>
<div class="paragraph">
<p>Note how our linker script also exposes some symbols to C:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lkmc_heap_low = .;
lkmc_heap_top = .;</pre>
</div>
</div>
<div class="paragraph">
<p>Those for example are required to implement <code>malloc</code> in Newlib. We can play with those variables more explicitly with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/linker_variables.c">baremetal/linker_variables.c</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/linker_variables.c</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-command-line-arguments"><a class="anchor" href="#baremetal-command-line-arguments"></a><a class="link" href="#baremetal-command-line-arguments">33.4. Baremetal command line arguments</a></h3>
<div class="paragraph">
<p>QEMU and gem5 currently supports baremetal CLI arguments!</p>
</div>
<div class="paragraph">
<p>You can see them in action e.g. with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal userland/c/command_line_arguments.c --cli-args 'aa bb cc'
./run --arch aarch64 --userland userland/c/command_line_arguments.c --cli-args 'aa bb cc'</pre>
</div>
</div>
<div class="paragraph">
<p>both of which output the exact same thing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>aa
bb
cc</pre>
</div>
</div>
<div class="paragraph">
<p>This is implemented by parsing the command line arguments and placing them into memory where the code will find them.</p>
</div>
<div class="paragraph">
<p>This works by:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>fixing the <code>argc</code> and <code>argv</code> addresses in memory in the <a href="#baremetal-linker-script">Baremetal linker script</a></p>
</li>
<li>
<p>the <a href="#baremetal-bootloaders">Baremetal bootloaders</a> pass those addresses correctly to the call of <code>main</code></p>
</li>
<li>
<p>our Python scripts write the desired binary memory values to a file</p>
</li>
<li>
<p>QEMU loads those files into memory with <code>-device loader</code>: <a href="https://github.com/qemu/qemu/blob/60905286cb5150de854e08279bca7dfc4b549e91/docs/generic-loader.txt" class="bare">https://github.com/qemu/qemu/blob/60905286cb5150de854e08279bca7dfc4b549e91/docs/generic-loader.txt</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>It is worth noting that e.g. ARM has a <a href="#semihosting">Semihosting</a> mechanism for loading CLI arguments through <code>SYS_GET_CMDLINE</code>, but our mechanism works in principle for any ISA.</p>
</div>
<div class="sect3">
<h4 id="gem5-baremetal-arm-cli-args"><a class="anchor" href="#gem5-baremetal-arm-cli-args"></a><a class="link" href="#gem5-baremetal-arm-cli-args">33.4.1. gem5 baremetal arm CLI args</a></h4>
<div class="paragraph">
<p>Currently not supported, so we just hardcode argc 0 on the <a href="#baremetal-bootloaders">arm baremetal bootloader</a>.</p>
</div>
<div class="paragraph">
<p>I think we have to keep the CLI args below 32 GiB, otherwise argc cannot be correctly setup. But currently the gem5 text segment is exactly at 32 GiB, and we always place the CLI args higher in the <a href="#baremetal-linker-script">Baremetal linker script</a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="semihosting"><a class="anchor" href="#semihosting"></a><a class="link" href="#semihosting">33.5. Semihosting</a></h3>
<div class="paragraph">
<p>Semihosting is a publicly documented interface specified by ARM Holdings that allows us to do some magic operations very useful in development, such as writting to the terminal or reading and writing host files.</p>
</div>
<div class="paragraph">
<p>It is documented at: <a href="https://developer.arm.com/docs/100863/latest/introduction" class="bare">https://developer.arm.com/docs/100863/latest/introduction</a></p>
</div>
<div class="paragraph">
<p>For example, all the following code make QEMU exit:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal baremetal/arch/arm/semihost_exit.S
./run --arch arm --baremetal baremetal/arch/arm/no_bootloader/semihost_exit.S
./run --arch aarch64 --baremetal baremetal/arch/aarch64/semihost_exit.S
./run --arch aarch64 --baremetal baremetal/arch/aarch64/no_bootloader/semihost_exit.S</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/semihost_exit.S">baremetal/arch/arm/semihost_exit.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/no_bootloader/semihost_exit.S">baremetal/arch/arm/no_bootloader/semihost_exit.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/semihost_exit.S">baremetal/arch/aarch64/semihost_exit.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/no_bootloader/semihost_exit.S">baremetal/arch/aarch64/no_bootloader/semihost_exit.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>That <code>arm</code> program program contains the code:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov r0, #0x18
ldr r1, =#0x20026
svc 0x00123456</pre>
</div>
</div>
<div class="paragraph">
<p>and we can see from the docs that <code>0x18</code> stands for the <code>SYS_EXIT</code> command.</p>
</div>
<div class="paragraph">
<p>This is also how we implement the <code>exit(0)</code> system call in C for QEMU, which is used for example at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit0.c">userland/c/exit0.c</a> through the Newlib via the <code>_exit</code> function at <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/syscalls.c">baremetal/lib/syscalls.c</a>.</p>
</div>
<div class="paragraph">
<p>Other magic operations we can do with semihosting besides exiting the on the host include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>read and write to host stdin and stdout</p>
</li>
<li>
<p>read and write to host files</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Alternatives exist for some semihosting operations, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>UART IO for host stdin and stdout in both emulators and real hardware</p>
</li>
<li>
<p><a href="#m5ops">m5ops</a> for <a href="#gem5">gem5</a>, e.g. <code>m5 exit</code> makes the emulator quit</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The big advantage of semihosting is that it is standardized across all ARM boards, and therefore allows you to make a single image that does those magic operations instead of having to compile multiple images with different magic addresses.</p>
</div>
<div class="paragraph">
<p>The downside of semihosting is that it is ARM specific. TODO is it an open standard that other vendors can implement?</p>
</div>
<div class="paragraph">
<p>In QEMU, we enable semihosting with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-semihosting</pre>
</div>
</div>
<div class="paragraph">
<p>Newlib 9c84bfd47922aad4881f80243320422b621c95dc already has a semi-hosting implementation at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>newlib/libc/sys/arm/syscalls.c</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: how to use it? Possible through crosstool-NG? In the worst case we could just copy it.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/31990487/how-to-cleanly-exit-qemu-after-executing-bare-metal-program-without-user-interve/40957928#40957928" class="bare">https://stackoverflow.com/questions/31990487/how-to-cleanly-exit-qemu-after-executing-bare-metal-program-without-user-interve/40957928#40957928</a></p>
</li>
<li>
<p><a href="https://balau82.wordpress.com/2010/11/04/qemu-arm-semihosting/" class="bare">https://balau82.wordpress.com/2010/11/04/qemu-arm-semihosting/</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gem5-semihosting"><a class="anchor" href="#gem5-semihosting"></a><a class="link" href="#gem5-semihosting">33.5.1. gem5 semihosting</a></h4>
<div class="paragraph">
<p>For gem5, you need <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/patches/manual/gem5-semihost.patch">patches/manual/gem5-semihost.patch</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>patch -d "$(./getvar gem5_source_dir)" -p 1 &lt; patches/manual/gem5-semihost.patch</pre>
</div>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/52475268/how-to-enable-arm-semihosting-in-gem5/52475269#52475269" class="bare">https://stackoverflow.com/questions/52475268/how-to-enable-arm-semihosting-in-gem5/52475269#52475269</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="gem5-baremetal-carriage-return"><a class="anchor" href="#gem5-baremetal-carriage-return"></a><a class="link" href="#gem5-baremetal-carriage-return">33.6. gem5 baremetal carriage return</a></h3>
<div class="paragraph">
<p>TODO: our example is printing newlines without automatic carriage return <code>\r</code> as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>enter a character
                 got: a</pre>
</div>
</div>
<div class="paragraph">
<p>We use <code>m5term</code> by default, and if we try <code>telnet</code> instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>telnet localhost 3456</pre>
</div>
</div>
<div class="paragraph">
<p>it does add the carriage returns automatically.</p>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-host-packaged-toolchain"><a class="anchor" href="#baremetal-host-packaged-toolchain"></a><a class="link" href="#baremetal-host-packaged-toolchain">33.7. Baremetal host packaged toolchain</a></h3>
<div class="paragraph">
<p>For <code>arm</code>, some baremetal examples compile fine with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>sudo apt-get install gcc-arm-none-eabi qemu-system-arm
./build-baremetal --arch arm --gcc-which host-baremetal
./run --arch arm --baremetal userland/c/hello.c --qemu-which host</pre>
</div>
</div>
<div class="paragraph">
<p>However, there are as usual limitations to using prebuilts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>certain examples fail to build with the Ubuntu packaged toolchain. E.g.: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit0.c">userland/c/exit0.c</a> fails with:</p>
<div class="literalblock">
<div class="content">
<pre>/usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/lib/libg.a(lib_a-fini.o): In function `__libc_fini_array':
/build/newlib-8gJlYR/newlib-2.4.0.20160527/build/arm-none-eabi/newlib/libc/misc/../../../../../newlib/libc/misc/fini.c:33: undefined reference to `_fini'
collect2: error: ld returned 1 exit status</pre>
</div>
</div>
<div class="paragraph">
<p>with the prebuilt toolchain, and I&#8217;m lazy to debug.</p>
</div>
</li>
<li>
<p>there seems to to be no analogous <code>aarch64</code> Ubuntu package to <code>gcc-arm-none-eabi</code>: <a href="https://askubuntu.com/questions/1049249/is-there-a-package-with-the-aarch64-version-of-gcc-arm-none-eabi-for-bare-metal" class="bare">https://askubuntu.com/questions/1049249/is-there-a-package-with-the-aarch64-version-of-gcc-arm-none-eabi-for-bare-metal</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-cpp"><a class="anchor" href="#baremetal-cpp"></a><a class="link" href="#baremetal-cpp">33.8. Baremetal C++</a></h3>
<div class="paragraph">
<p>Didn&#8217;t get it working, traking at: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/119" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/119</a></p>
</div>
</div>
<div class="sect2">
<h3 id="gdb-builtin-cpu-simulator"><a class="anchor" href="#gdb-builtin-cpu-simulator"></a><a class="link" href="#gdb-builtin-cpu-simulator">33.9. GDB builtin CPU simulator</a></h3>
<div class="paragraph">
<p>It is incredible, but GDB also has a CPU simulator inside of it as documented at: <a href="https://sourceware.org/gdb/onlinedocs/gdb/Target-Commands.html" class="bare">https://sourceware.org/gdb/onlinedocs/gdb/Target-Commands.html</a></p>
</div>
<div class="paragraph">
<p>TODO: any advantage over QEMU? I doubt it, mostly using it as as toy for now:</p>
</div>
<div class="paragraph">
<p>Without running <code>./run</code>, do directly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm --baremetal userland/c/hello.c --sim</pre>
</div>
</div>
<div class="paragraph">
<p>Then inside GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>load
starti</pre>
</div>
</div>
<div class="paragraph">
<p>and now you can debug normally.</p>
</div>
<div class="paragraph">
<p>Enabled with the crosstool-NG configuration:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CT_GDB_CROSS_SIM=y</pre>
</div>
</div>
<div class="paragraph">
<p>which by grepping crosstool-NG we can see does on GDB:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./configure --enable-sim</pre>
</div>
</div>
<div class="paragraph">
<p>Those are not set by default on <code>gdb-multiarch</code> in Ubuntu 16.04.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/49470659/arm-none-eabi-gdb-undefined-target-command-sim" class="bare">https://stackoverflow.com/questions/49470659/arm-none-eabi-gdb-undefined-target-command-sim</a></p>
</li>
<li>
<p><a href="http://cs107e.github.io/guides/gdb/" class="bare">http://cs107e.github.io/guides/gdb/</a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="gdb-builtin-cpu-simulator-userland"><a class="anchor" href="#gdb-builtin-cpu-simulator-userland"></a><a class="link" href="#gdb-builtin-cpu-simulator-userland">33.9.1. GDB builtin CPU simulator userland</a></h4>
<div class="paragraph">
<p>Since I had this compiled, I also decided to try it out on userland.</p>
</div>
<div class="paragraph">
<p>I was also able to run a freestanding Linux userland example on it: <a href="https://github.com/cirosantilli/arm-assembly-cheat/blob/cd232dcaf32c0ba6399b407e0b143d19b6ec15f4/v7/linux/hello.S" class="bare">https://github.com/cirosantilli/arm-assembly-cheat/blob/cd232dcaf32c0ba6399b407e0b143d19b6ec15f4/v7/linux/hello.S</a></p>
</div>
<div class="paragraph">
<p>It just ignores the <a href="#arm-svc-instruction">ARM SVC instruction</a> however, and does not forward syscalls to the host like QEMU does.</p>
</div>
<div class="paragraph">
<p>Then I tried a glibc example: <a href="https://github.com/cirosantilli/arm-assembly-cheat/blob/cd232dcaf32c0ba6399b407e0b143d19b6ec15f4/v7/mov.S" class="bare">https://github.com/cirosantilli/arm-assembly-cheat/blob/cd232dcaf32c0ba6399b407e0b143d19b6ec15f4/v7/mov.S</a></p>
</div>
<div class="paragraph">
<p>First it wouldn&#8217;t break, so I added <code>-static</code> to the <code>Makefile</code>, and then it started failing with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Unhandled v6 thumb insn</pre>
</div>
</div>
<div class="paragraph">
<p>Doing:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>help architecture</pre>
</div>
</div>
<div class="paragraph">
<p>shows ARM version up to <code>armv6</code>, so maybe <code>armv6</code> is not implemented?</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="arm-baremetal"><a class="anchor" href="#arm-baremetal"></a><a class="link" href="#arm-baremetal">33.10. ARM baremetal</a></h3>
<div class="paragraph">
<p>In this section we will focus on learning ARM architecture concepts that can only learnt on baremetal setups.</p>
</div>
<div class="paragraph">
<p>Userland information can be found at: <a href="https://github.com/cirosantilli/arm-assembly-cheat" class="bare">https://github.com/cirosantilli/arm-assembly-cheat</a></p>
</div>
<div class="sect3">
<h4 id="arm-exception-levels"><a class="anchor" href="#arm-exception-levels"></a><a class="link" href="#arm-exception-levels">33.10.1. ARM exception levels</a></h4>
<div class="paragraph">
<p>ARM exception levels are analogous to x86 <a href="#ring0">rings</a>.</p>
</div>
<div class="paragraph">
<p>The current EL can be determined by reading from certain registers, which we do with bit disassembly at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/arch/arm/dump_regs.c
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c</pre>
</div>
</div>
<div class="paragraph">
<p>The relevant bits are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>arm: <code>CPSR.M</code></p>
</li>
<li>
<p>aarch64: <code>CurrentEl.EL</code>. This register is not accessible from EL0 for some weird reason however.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/dump_regs.c">baremetal/arch/arm/dump_regs.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/dump_regs.c">baremetal/arch/aarch64/dump_regs.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The instructions that find the ARM EL are explained at: <a href="https://stackoverflow.com/questions/31787617/what-is-the-current-execution-mode-exception-level-etc" class="bare">https://stackoverflow.com/questions/31787617/what-is-the-current-execution-mode-exception-level-etc</a></p>
</div>
<div class="paragraph">
<p>The lower ELs are not mandated by the architecture, and can be controlled through command line options in QEMU and gem5.</p>
</div>
<div class="paragraph">
<p>In QEMU, you can configure the lowest EL as explained at <a href="https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up" class="bare">https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/arch/arm/dump_regs.c | grep CPSR.M
./run --arch arm --baremetal userland/arch/arm/dump_regs.c -- -machine virtualization=on | grep CPSR.M
./run --arch arm --baremetal userland/arch/arm/dump_regs.c -- -machine secure=on | grep CPSR.M
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c | grep CurrentEL.EL
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c -- -machine virtualization=on | grep CurrentEL.EL
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c -- -machine secure=on | grep CurrentEL.EL</pre>
</div>
</div>
<div class="paragraph">
<p>outputs respectively:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CPSR.M 0x3
CPSR.M 0x3
CPSR.M 0x3
CurrentEL.EL 0x1
CurrentEL.EL 0x2
CurrentEL.EL 0x3</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: why is arm <code>CPSR.M</code> stuck at <code>0x3</code> which equals Supervisor mode?</p>
</div>
<div class="paragraph">
<p>In gem5, you can configure the lowest EL with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/arch/arm/dump_regs.c --emulator gem5
grep CPSR.M "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch arm --baremetal userland/arch/arm/dump_regs.c --emulator gem5 -- --param 'system.have_virtualization = True'
grep CPSR.M "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch arm --baremetal userland/arch/arm/dump_regs.c --emulator gem5 -- --param 'system.have_security = True'
grep CPSR.M "$(./getvar --arch arm --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c --emulator gem5
grep CurrentEL.EL "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c --emulator gem5 -- --param 'system.have_virtualization = True'
grep CurrentEL.EL "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c --emulator gem5 -- --param 'system.have_security = True'
grep CurrentEL.EL "$(./getvar --arch aarch64 --emulator gem5 gem5_guest_terminal_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>CPSR.M 0x3
CPSR.M 0xA
CPSR.M 0x3
CurrentEL.EL 0x1
CurrentEL.EL 0x2
CurrentEL.EL 0x3</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: the call:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --baremetal userland/arch/arm/dump_regs.c --emulator gem5 -- --param 'system.have_virtualization = True'</pre>
</div>
</div>
<div class="paragraph">
<p>started failing with an exception since <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/commit/add6eedb76636b8f443b815c6b2dd160afdb7ff4" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/commit/add6eedb76636b8f443b815c6b2dd160afdb7ff4</a> at the instruction:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vmsr fpexc, r0</pre>
</div>
</div>
<div class="paragraph">
<p>in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/arm.S">baremetal/lib/arm.S</a>. That patch however enables SIMD in baremetal, which I feel is more important.</p>
</div>
<div class="paragraph">
<p>According to <a href="#armarm7">ARMv7 architecture reference manual</a>, access to that register is controlled by other registers <code>NSACR.{CP11, CP10}</code> and <code>HCPTR</code> so those must be turned off, but I&#8217;m lazy to investigate now, even just trying to dump those registers in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/dump_regs.c">userland/arch/arm/dump_regs.c</a> also leads to exceptions&#8230;&#8203;</p>
</div>
<div class="sect4">
<h5 id="arm-change-exception-level"><a class="anchor" href="#arm-change-exception-level"></a><a class="link" href="#arm-change-exception-level">33.10.1.1. ARM change exception level</a></h5>
<div class="paragraph">
<p>TODO. Create a minimal runnable example of going into EL0 and jumping to EL1.</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-sp0-vs-spx"><a class="anchor" href="#arm-sp0-vs-spx"></a><a class="link" href="#arm-sp0-vs-spx">33.10.1.2. ARM SP0 vs SPx</a></h5>
<div class="paragraph">
<p>See <a href="#armarm8-db">ARMv8 architecture reference manual db</a> D1.6.2 "The stack pointer registers".</p>
</div>
<div class="paragraph">
<p>There is one SP per <a href="#arm-exception-levels">exception level</a>.</p>
</div>
<div class="paragraph">
<p>This can also be seen clearly on the analysis at <a href="#gem5-execcontext-readintregoperand-register-resolution">gem5 <code>ExecContext::readIntRegOperand</code> register resolution</a>.</p>
</div>
<div class="paragraph">
<p>TODO create a minimal runnable example.</p>
</div>
<div class="paragraph">
<p>TODO: how to select to use SP0 in an exception handler?</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-svc-instruction"><a class="anchor" href="#arm-svc-instruction"></a><a class="link" href="#arm-svc-instruction">33.10.2. ARM SVC instruction</a></h4>
<div class="paragraph">
<p>This is the most basic example of exception handling we have.</p>
</div>
<div class="paragraph">
<p>We a handler for SVC, do an SVC, and observe that the handler got called and returned from C and assembly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/svc.c
./run --arch aarch64 --baremetal baremetal/arch/aarch64/svc_asm.S</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/svc.c">baremetal/arch/aarch64/svc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/svc_asm.S">baremetal/arch/aarch64/svc_asm.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Sample output for the C one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>DAIF 0x3C0
SPSEL 0x1
VBAR_EL1 0x40000800
after_svc 0x4000209c
lkmc_vector_trap_handler
exc_type 0x11
exc_type is LKMC_VECTOR_SYNC_SPX
ESR 0x5600ABCD
ESR.EC 0x15
ESR.EC.ISS.imm16 0xABCD
SP 0x4200C510
ELR 0x4000209C
SPSR 0x600003C5
x0 0x0
x1 0x1
x2 0x15
x3 0x15
x4 0x4000A178
x5 0xFFFFFFF6
x6 0x4200C390
x7 0x78
x8 0x1
x9 0x14
x10 0x0
x11 0x0
x12 0x0
x13 0x0
x14 0x0
x15 0x0
x16 0x0
x17 0x0
x18 0x0
x19 0x0
x20 0x0
x21 0x0
x22 0x0
x23 0x0
x24 0x0
x25 0x0
x26 0x0
x27 0x0
x28 0x0
x29 0x4200C510
x30 0x40002064</pre>
</div>
</div>
<div class="paragraph">
<p>The C code does an:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>svc 0xABCD</pre>
</div>
</div>
<div class="paragraph">
<p>and the value 0xABCD appears at the bottom of <a href="#arm-esr-register">ARM ESR register</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ESR 0x5600ABCD
ESR.EC 0x15
ESR.EC.ISS.imm16 0xABCD</pre>
</div>
</div>
<div class="paragraph">
<p>The other important register is the <a href="#arm-elr-register">ARM ELR register</a>, which contains the return address after the exception.</p>
</div>
<div class="paragraph">
<p>From the output, we can see that it matches the value as obtained by taking the address of a label placed just after the SVC:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>after_svc 0x4000209c
ELR 0x4000209C</pre>
</div>
</div>
<div class="paragraph">
<p>Both QEMU and gem5 are able to trace interrupts in addition to instructions, and it is instructive to enable both and have a look at the traces.</p>
</div>
<div class="paragraph">
<p>With <a href="#qemu-d-tracing">QEMU -d tracing</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --baremetal baremetal/arch/aarch64/svc.c \
  -- -d in_asm,int \
;</pre>
</div>
</div>
<div class="paragraph">
<p>the output at 8f73910dd1fc1fa6dc6904ae406b7598cdcd96d7 contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>----------------
IN: main
0x40002098:  d41579a1  svc      #0xabcd

Taking exception 2 [SVC]
...from EL1 to EL1
...with ESR 0x15/0x5600abcd
...with ELR 0x4000209c
...to EL1 PC 0x40000a00 PSTATE 0x3c5
----------------
IN:
0x40000a00:  14000225  b        #0x40001294

----------------
IN:
0x40001294:  a9bf7bfd  stp      x29, x30, [sp, #-0x10]!
0x40001298:  a9bf73fb  stp      x27, x28, [sp, #-0x10]!
0x4000129c:  a9bf6bf9  stp      x25, x26, [sp, #-0x10]!
0x400012a0:  a9bf63f7  stp      x23, x24, [sp, #-0x10]!
0x400012a4:  a9bf5bf5  stp      x21, x22, [sp, #-0x10]!
0x400012a8:  a9bf53f3  stp      x19, x20, [sp, #-0x10]!
0x400012ac:  a9bf4bf1  stp      x17, x18, [sp, #-0x10]!
0x400012b0:  a9bf43ef  stp      x15, x16, [sp, #-0x10]!
0x400012b4:  a9bf3bed  stp      x13, x14, [sp, #-0x10]!
0x400012b8:  a9bf33eb  stp      x11, x12, [sp, #-0x10]!
0x400012bc:  a9bf2be9  stp      x9, x10, [sp, #-0x10]!
0x400012c0:  a9bf23e7  stp      x7, x8, [sp, #-0x10]!
0x400012c4:  a9bf1be5  stp      x5, x6, [sp, #-0x10]!
0x400012c8:  a9bf13e3  stp      x3, x4, [sp, #-0x10]!
0x400012cc:  a9bf0be1  stp      x1, x2, [sp, #-0x10]!
0x400012d0:  d5384015  mrs      x21, spsr_el1
0x400012d4:  a9bf03f5  stp      x21, x0, [sp, #-0x10]!
0x400012d8:  d5384035  mrs      x21, elr_el1
0x400012dc:  a9bf57ff  stp      xzr, x21, [sp, #-0x10]!
0x400012e0:  d2800235  movz     x21, #0x11
0x400012e4:  d5385216  mrs      x22, esr_el1
0x400012e8:  a9bf5bf5  stp      x21, x22, [sp, #-0x10]!
0x400012ec:  910003f5  mov      x21, sp
0x400012f0:  910482b5  add      x21, x21, #0x120
0x400012f4:  f9000bf5  str      x21, [sp, #0x10]
0x400012f8:  910003e0  mov      x0, sp
0x400012fc:  9400023f  bl       #0x40001bf8

----------------
IN: lkmc_vector_trap_handler
0x40001bf8:  a9bd7bfd  stp      x29, x30, [sp, #-0x30]!</pre>
</div>
</div>
<div class="paragraph">
<p>And with <a href="#gem5-tracing">gem5 tracing</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch aarch64 \
  --baremetal baremetal/arch/aarch64/svc_asm.S \
  --trace ExecAll,Faults \
  --trace-stdout \
;</pre>
</div>
</div>
<div class="paragraph">
<p>the output contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   4000: system.cpu A0 T0 : @main+8    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)
   4000: Supervisor Call: Invoking Fault (AArch64 target EL):Supervisor Call cpsr:0x3c5 PC:0x80000808 elr:0x8000080c newVec: 0x80001200
   4500: system.cpu A0 T0 : @vector_table+512    :   b   &lt;_curr_el_spx_sync&gt;  : IntAlu :   flags=(IsControl|IsDirectControl|IsUncondControl)</pre>
</div>
</div>
<div class="paragraph">
<p>So we see in both cases that the:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>SVC is done</p>
</li>
<li>
<p>an exception happens, and the PC jumps to address 0x40000a00. From our custom terminal prints further on, we see that this equals <code>VBAR_EL1 + 0x200</code>.</p>
<div class="paragraph">
<p>According to the format of the <a href="#armv8-exception-vector-table-format">ARMv8 exception vector table format</a>, we see that the <code>+ 0x200</code> means that we are jumping in the Current EL with SPx.</p>
</div>
<div class="paragraph">
<p>This can also be deduced from the message <code>exc_type is LKMC_VECTOR_SYNC_SPX</code>: we just manually store a different integer for every exception vector type in our handler code to be able to tell what happened.</p>
</div>
<div class="paragraph">
<p>This is the one used because we are jumping <a href="#arm-exception-levels">from EL1 to EL1</a>.</p>
</div>
<div class="paragraph">
<p>We set VBAR_EL1 to that address ourselves <a href="#baremetal-bootloaders">in the bootloader</a>.</p>
</div>
</li>
<li>
<p>at 0x40000a00 a <code>b #0x40001294</code> is done and then at 0x40001294 boilerplate preparation is done for lkmc_vector_trap_handler starting with several STP instructions.</p>
<div class="paragraph">
<p>We have coded both of those in our vector table macro madness. As of LKMC 8f73910dd1fc1fa6dc6904ae406b7598cdcd96d7, both come from <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/aarch64.h">lkmc/aarch64.h</a>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>b #0x40001294</code> comes from: <code>LKMC_VECTOR_ENTRY</code></p>
</li>
<li>
<p>the STP come from: <code>LKMC_VECTOR_BUILD_TRAPFRAME</code></p>
<div class="paragraph">
<p>We jump immediately from inside <code>LKMC_VECTOR_ENTRY</code> to <code>LKMC_VECTOR_BUILD_TRAPFRAME</code> because we can only use 0x80 bytes of instructions for each one before reaching the next handler, so we might as well get it over with by jumping into a memory region without those constraints.</p>
</div>
<div class="paragraph">
<p>TODO: why doesn&#8217;t QEMU show our nice symbol names? gem5 shows them fine, and <code>nm</code> says they are there!</p>
</div>
<div class="literalblock">
<div class="content">
<pre>0000000040000800 T lkmc_vector_table
0000000040001294 T lkmc_vector_build_trapframe_curr_el_spx_sync</pre>
</div>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>The exception return happens at the end of <code>lkmc_vector_trap_handler</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>----------------
IN: lkmc_vector_trap_handler
0x40002000:  d503201f  nop
0x40002004:  a8c37bfd  ldp      x29, x30, [sp], #0x30
0x40002008:  d65f03c0  ret

----------------
IN:
0x40001300:  910043ff  add      sp, sp, #0x10
0x40001304:  a8c15bf5  ldp      x21, x22, [sp], #0x10
0x40001308:  d5184036  msr      elr_el1, x22

----------------
IN:
0x4000130c:  a8c103f5  ldp      x21, x0, [sp], #0x10
0x40001310:  d5184015  msr      spsr_el1, x21

----------------
IN:
0x40001314:  a8c10be1  ldp      x1, x2, [sp], #0x10
0x40001318:  a8c113e3  ldp      x3, x4, [sp], #0x10
0x4000131c:  a8c11be5  ldp      x5, x6, [sp], #0x10
0x40001320:  a8c123e7  ldp      x7, x8, [sp], #0x10
0x40001324:  a8c12be9  ldp      x9, x10, [sp], #0x10
0x40001328:  a8c133eb  ldp      x11, x12, [sp], #0x10
0x4000132c:  a8c13bed  ldp      x13, x14, [sp], #0x10
0x40001330:  a8c143ef  ldp      x15, x16, [sp], #0x10
0x40001334:  a8c14bf1  ldp      x17, x18, [sp], #0x10
0x40001338:  a8c153f3  ldp      x19, x20, [sp], #0x10
0x4000133c:  a8c15bf5  ldp      x21, x22, [sp], #0x10
0x40001340:  a8c163f7  ldp      x23, x24, [sp], #0x10
0x40001344:  a8c16bf9  ldp      x25, x26, [sp], #0x10
0x40001348:  a8c173fb  ldp      x27, x28, [sp], #0x10
0x4000134c:  a8c17bfd  ldp      x29, x30, [sp], #0x10
0x40001350:  d69f03e0  eret

Exception return from AArch64 EL1 to AArch64 EL1 PC 0x4000209c
----------------
IN: main
0x4000209c:  d0000040  adrp     x0, #0x4000c000</pre>
</div>
</div>
<div class="paragraph">
<p>which does an <code>eret</code> and jumps back to 0x4000209c, which is 4 bytes and therefore one instruction after where SVC was taken at 0x40002098.</p>
</div>
<div class="paragraph">
<p>On the terminal output, we observe the initial values of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>DAIF: 0x3c0, i.e. 4 bits (6 to 9) set to 1, which means that exceptions are masked for each exception type: Synchronous, System error, IRQ and FIQ.</p>
<div class="paragraph">
<p>This reset value is defined by <a href="#armarm8">ARMv8 architecture reference manual</a> C5.2.2 "DAIF, Interrupt Mask Bits".</p>
</div>
</li>
<li>
<p>SPSel: 0x1, which means: use SPx instead of SP0.</p>
<div class="paragraph">
<p>This reset value is defined by <a href="#armarm8">ARMv8 architecture reference manual</a> C5.2.16 "SPSel, Stack Pointer Select".</p>
</div>
</li>
<li>
<p>VBAR_EL1: 0x0 holds the base address of the vector table</p>
<div class="paragraph">
<p>This reset value is defined UNKNOWN by <a href="#armarm8">ARMv8 architecture reference manual</a> D10.2.116 "VBAR_EL1, Vector Base Address Register (EL1)", so we must set it to something ourselves to have greater portability.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/torvalds/linux/blob/v4.20/arch/arm64/kernel/entry.S#L430" class="bare">https://github.com/torvalds/linux/blob/v4.20/arch/arm64/kernel/entry.S#L430</a> this is where the kernel defines the vector table</p>
</li>
<li>
<p><a href="https://github.com/dwelch67/qemu_arm_samples/tree/07162ba087111e0df3f44fd857d1b4e82458a56d/swi01" class="bare">https://github.com/dwelch67/qemu_arm_samples/tree/07162ba087111e0df3f44fd857d1b4e82458a56d/swi01</a></p>
</li>
<li>
<p><a href="https://github.com/NienfengYao/armv8-bare-metal/blob/572c6f95880e70aa92fe9fed4b8ad7697082a764/vector.S#L168" class="bare">https://github.com/NienfengYao/armv8-bare-metal/blob/572c6f95880e70aa92fe9fed4b8ad7697082a764/vector.S#L168</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/24162109/arm-assembly-code-and-svc-numbering/57064062#57064062" class="bare">https://stackoverflow.com/questions/24162109/arm-assembly-code-and-svc-numbering/57064062#57064062</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/44991264/armv8-exception-vectors-and-handling" class="bare">https://stackoverflow.com/questions/44991264/armv8-exception-vectors-and-handling</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="armv8-exception-vector-table-format"><a class="anchor" href="#armv8-exception-vector-table-format"></a><a class="link" href="#armv8-exception-vector-table-format">33.10.2.1. ARMv8 exception vector table format</a></h5>
<div class="paragraph">
<p>The vector table format is described on <a href="#armarm8">ARMv8 architecture reference manual</a> Table D1-7 "Vector offsets from vector table base address".</p>
</div>
<div class="paragraph">
<p>A good representation of the format of the vector table can also be found at <a href="#armv8-programmers-guide">Programmer&#8217;s Guide for ARMv8-A</a> Table 10-2 "Vector table offsets from vector table base address".</p>
</div>
<div class="paragraph">
<p>The first part of the table contains: <a href="#table-armv8-vector-handlers">Table 6, &#8220;Summary of ARMv8 vector handlers&#8221;</a>.</p>
</div>
<table id="table-armv8-vector-handlers" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 6. Summary of ARMv8 vector handlers</caption>
<colgroup>
<col style="width: 33.3333%;">
<col style="width: 33.3333%;">
<col style="width: 33.3334%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Address</th>
<th class="tableblock halign-left valign-top">Exception type</th>
<th class="tableblock halign-left valign-top">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x000</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Synchronous</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SP0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x080</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">IRQ/vIRQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SP0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x100</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">FIQ/vFIQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SP0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x180</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SError/vSError</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SP0</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x200</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Synchronous</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SPx</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x280</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">IRQ/vIRQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SPx</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x300</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">FIQ/vFIQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Current EL with SPx</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x380</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SError/vSError</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch64</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x400</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Synchronous</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch64</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x480</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">IRQ/vIRQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch64</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x500</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">FIQ/vFIQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch64</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x580</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SError/vSError</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch64</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x600</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Synchronous</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch32</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x680</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">IRQ/vIRQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch32</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x700</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">FIQ/vFIQ</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch32</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">VBAR_ELn + 0x780</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">SError/vSError</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Lower EL using AArch32</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>and the following other parts are analogous, but referring to SPx and lower ELs.</p>
</div>
<div class="paragraph">
<p>Now, to fully understand this table, we need the following concepts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Synchronous: what happens for example when we do an <a href="#arm-svc-instruction">ARM SVC instruction</a>.</p>
<div class="paragraph">
<p>It is called synchronous because the CPU is generating it itself from an instruction, unlike an interrupt generated by a device like a keyboard, which ends up in an IRQ or FIQ</p>
</div>
</li>
<li>
<p>IRQ: an example can be found at: <a href="#arm-timer">ARM timer</a></p>
</li>
<li>
<p>TODO FIQ vs IRQ</p>
</li>
<li>
<p>TODO SError</p>
</li>
<li>
<p>EL changes: <a href="#arm-change-exception-level">ARM change exception level</a></p>
</li>
<li>
<p>SP0 vs SPx: <a href="#arm-sp0-vs-spx">ARM SP0 vs SPx</a>.</p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="arm-esr-register"><a class="anchor" href="#arm-esr-register"></a><a class="link" href="#arm-esr-register">33.10.2.2. ARM ESR register</a></h5>
<div class="paragraph">
<p>Exception Syndrome Register.</p>
</div>
<div class="paragraph">
<p>See example at: <a href="#arm-svc-instruction">Section 33.10.2, &#8220;ARM SVC instruction&#8221;</a></p>
</div>
<div class="paragraph">
<p>Documentation: <a href="#armarm8-db">ARMv8 architecture reference manual db</a> D12.2.36 "ESR_EL1, Exception Syndrome Register (EL1)".</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-elr-register"><a class="anchor" href="#arm-elr-register"></a><a class="link" href="#arm-elr-register">33.10.2.3. ARM ELR register</a></h5>
<div class="paragraph">
<p>Exception Link Register.</p>
</div>
<div class="paragraph">
<p>See the example at: <a href="#arm-svc-instruction">Section 33.10.2, &#8220;ARM SVC instruction&#8221;</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-baremetal-multicore"><a class="anchor" href="#arm-baremetal-multicore"></a><a class="link" href="#arm-baremetal-multicore">33.10.3. ARM baremetal multicore</a></h4>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/no_bootloader/multicore_asm.S --cpus 2
./run --arch aarch64 --baremetal baremetal/arch/aarch64/no_bootloader/multicore_asm.S --cpus 2 --emulator gem5
./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.c --cpus 2
./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.c --cpus 2 --emulator gem5
./run --arch arm --baremetal baremetal/arch/arm/no_bootloader/multicore_asm.S --cpus 2
./run --arch arm --baremetal baremetal/arch/arm/no_bootloader/multicore_asm.S --cpus 2 --emulator gem5
# TODO not working, hangs.
# ./run --arch arm --baremetal baremetal/arch/arm/multicore.c --cpus 2
./run --arch arm --baremetal baremetal/arch/arm/multicore.c --cpus 2 --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/no_bootloader/multicore_asm.S">baremetal/arch/aarch64/no_bootloader/multicore_asm.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/multicore.c">baremetal/arch/aarch64/multicore.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/no_bootloader/multicore_asm.S">baremetal/arch/arm/no_bootloader/multicore_asm.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/multicore.c">baremetal/arch/arm/multicore.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>CPU 0 of this program enters a spinlock loop: it repeatedly checks if a given memory address is 1.</p>
</div>
<div class="paragraph">
<p>So, we need CPU 1 to come to the rescue and set that memory address to 1, otherwise CPU 0 will be stuck there forever!</p>
</div>
<div class="paragraph">
<p>Don&#8217;t believe me? Then try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.c --cpus 1</pre>
</div>
</div>
<div class="paragraph">
<p>and watch it hang forever.</p>
</div>
<div class="paragraph">
<p>Note that if you try the same thing on gem5:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/multicore.c --cpus 1 --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>then the gem5 actually exits with <a href="#gem5-simulate-limit-reached">gem5 simulate() limit reached</a> as opposed to the expected:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Exiting @ tick 36500 because m5_exit instruction encountered</pre>
</div>
</div>
<div class="paragraph">
<p>since gem5 is able to detect when nothing will ever happen, and exits.</p>
</div>
<div class="paragraph">
<p>When GDB step debugging, switch between cores with the usual <code>thread</code> commands, see also: <a href="#gdb-step-debug-multicore-userland">Section 3.9, &#8220;GDB step debug multicore userland&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438" class="bare">https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438</a></p>
</div>
<div class="sect4">
<h5 id="arm-wfe-and-sev-instructions"><a class="anchor" href="#arm-wfe-and-sev-instructions"></a><a class="link" href="#arm-wfe-and-sev-instructions">33.10.3.1. ARM WFE and SEV instructions</a></h5>
<div class="paragraph">
<p>The WFE and SEV instructions are just hints: a compliant implementation can treat them as NOPs.</p>
</div>
<div class="paragraph">
<p>Concrete examples of the instruction can be seen at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/nostartfiles/wfe.S">userland/arch/aarch64/nostartfiles/wfe.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/wfe.S">userland/arch/aarch64/freestanding/linux/wfe.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/sevl_wfe.S">userland/arch/aarch64/freestanding/linux/sevl_wfe.S</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/wfe_wfe.S">userland/arch/aarch64/freestanding/linux/wfe_wfe.S</a>: run WFE twice, because gem5 390a74f59934b85d91489f8a563450d8321b602d does not sleep on the first, see also: <a href="#gem5-arm-wfe">gem5 ARM WFE</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/no_bootloader/wfe_loop.S">baremetal/arch/aarch64/no_bootloader/wfe_loop.S</a>, see: <a href="#gem5-simulate-limit-reached">gem5 simulate() limit reached</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/wfe_sev.cpp">userland/arch/aarch64/inline_asm/wfe_sev.cpp</a>: one Linux thread runs WFE and the other runs SEV to wake it up</p>
</li>
<li>
<p><a href="#arm-baremetal-multicore">ARM baremetal multicore</a> shows baremetal examples where WFE sleeps and another thread wakes it up:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/multicore.c">baremetal/arch/arm/multicore.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/multicore.c">baremetal/arch/aarch64/multicore.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/arm/no_bootloader/multicore_asm.S">baremetal/arch/arm/no_bootloader/multicore_asm.S</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>However, likely no implementation likely does (TODO confirm), since:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>WFE is intended to put the core in a low power mode</p>
</li>
<li>
<p>SEV wakes up cores from a low power mode</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and power consumption is key in ARM applications.</p>
</div>
<div class="paragraph">
<p>Quotes for the above <a href="#armarm8-db">ARMv8 architecture reference manual db</a> G1.18.1 "Wait For Event and Send Event":</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>The following events are WFE wake-up events:</p>
</div>
<div class="paragraph">
<p>\[&#8230;&#8203;]</p>
</div>
<div class="ulist">
<ul>
<li>
<p>An event caused by the clearing of the global monitor associated with the PE</p>
</li>
</ul>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>and <a href="#armarm8-db">ARMv8 architecture reference manual db</a> E2.9.6 "Use of WFE and SEV instructions by spin-locks":</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>ARMv8 provides Wait For Event, Send Event, and Send Event Local instructions, WFE, SEV, SEVL, that can assist with reducing power consumption and bus contention caused by PEs repeatedly attempting to obtain a spin-lock. These instructions can be used at the application level, but a complete understanding of what they do depends on a system level understanding of exceptions. They are described in Wait For Event and Send Event on page G1-5308. However, in ARMv8, when the global monitor for a PE changes from Exclusive Access state to Open Access state, an event is generated.</p>
</div>
<div class="paragraph">
<p>Note This is equivalent to issuing an SEVL instruction on the PE for which the monitor state has changed. It removes the need for spinlock code to include an SEV instruction after clearing a spinlock.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>The recommended ARMv8 spinlock implementation is shown at <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/ch01s03s02.html" class="bare">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/ch01s03s02.html</a> where <code>WAIT_FOR_UPDATE</code> is as explained in that section a macro that expands to WFE. TODO SEV is used explicitly in those examples via SIGNAL_UPDATE, where is the example that shows how SEV can be eliminated due to implicit monitor signals?</p>
</div>
<div class="paragraph">
<p>In QEMU 3.0.0, SEV is a NOPs, and WFE might be, but I&#8217;m not sure, see: <a href="https://github.com/qemu/qemu/blob/v3.0.0/target/arm/translate-a64.c#L1423" class="bare">https://github.com/qemu/qemu/blob/v3.0.0/target/arm/translate-a64.c#L1423</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>    case 2: /* WFE */
        if (!(tb_cflags(s-&gt;base.tb) &amp; CF_PARALLEL)) {
            s-&gt;base.is_jmp = DISAS_WFE;
        }
        return;
    case 4: /* SEV */
    case 5: /* SEVL */
        /* we treat all as NOP at least for now */
        return;</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: what does the WFE code do? How can it not be a NOP if SEV is a NOP? <a href="https://github.com/qemu/qemu/blob/v3.0.0/target/arm/translate.c#L4609" class="bare">https://github.com/qemu/qemu/blob/v3.0.0/target/arm/translate.c#L4609</a> might explain why, but it is Chinese to me (I only understand 30% ;-)):</p>
</div>
<div class="literalblock">
<div class="content">
<pre> * For WFI we will halt the vCPU until an IRQ. For WFE and YIELD we
 * only call the helper when running single threaded TCG code to ensure
 * the next round-robin scheduled vCPU gets a crack. In MTTCG mode we
 * just skip this instruction. Currently the SEV/SEVL instructions
 * which are *one* of many ways to wake the CPU from WFE are not
 * implemented so we can't sleep like WFI does.
 */</pre>
</div>
</div>
<div class="paragraph">
<p>For gem5 however, if we comment out the SVE instruction, then it actually exits with <code>simulate() limit reached</code>, so the CPU truly never wakes up, which is a more realistic behaviour, since gem5 is more focused on simulating a realistic microarchitecture and power consumption.</p>
</div>
<div class="paragraph">
<p>The following Raspberry Pi bibliography helped us get this sample up and running:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/bztsrc/raspi3-tutorial/tree/a3f069b794aeebef633dbe1af3610784d55a0efa/02_multicorec" class="bare">https://github.com/bztsrc/raspi3-tutorial/tree/a3f069b794aeebef633dbe1af3610784d55a0efa/02_multicorec</a></p>
</li>
<li>
<p><a href="https://github.com/dwelch67/raspberrypi/tree/a09771a1d5a0b53d8e7a461948dc226c5467aeec/multi00" class="bare">https://github.com/dwelch67/raspberrypi/tree/a09771a1d5a0b53d8e7a461948dc226c5467aeec/multi00</a></p>
</li>
<li>
<p><a href="https://github.com/LdB-ECM/Raspberry-Pi/blob/3b628a2c113b3997ffdb408db03093b2953e4961/Multicore/SmartStart64.S" class="bare">https://github.com/LdB-ECM/Raspberry-Pi/blob/3b628a2c113b3997ffdb408db03093b2953e4961/Multicore/SmartStart64.S</a></p>
</li>
<li>
<p><a href="https://github.com/LdB-ECM/Raspberry-Pi/blob/3b628a2c113b3997ffdb408db03093b2953e4961/Multicore/SmartStart32.S" class="bare">https://github.com/LdB-ECM/Raspberry-Pi/blob/3b628a2c113b3997ffdb408db03093b2953e4961/Multicore/SmartStart32.S</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>For how userland spinlocks and mutexes are implemented see <a href="#userland-mutex-implementation">Userland mutex implementation</a>.</p>
</div>
<div class="sect5">
<h6 id="arm-wfe-global-monitor-events"><a class="anchor" href="#arm-wfe-global-monitor-events"></a><a class="link" href="#arm-wfe-global-monitor-events">33.10.3.1.1. ARM WFE global monitor events</a></h6>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/wfe_ldxr_stxr.cpp">userland/arch/aarch64/inline_asm/wfe_ldxr_stxr.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/wfe_ldxr_str.cpp">userland/arch/aarch64/inline_asm/wfe_ldxr_str.cpp</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/futex_ldxr_stxr.c">userland/arch/aarch64/inline_asm/futex_ldxr_stxr.c</a>: tests that ldxr and stxr do not interact with futexes. This was leading to problems in <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a> at one point: <a href="https://gem5.atlassian.net/browse/GEM5-537" class="bare">https://gem5.atlassian.net/browse/GEM5-537</a></p>
<div class="paragraph">
<p>Correct outcome: <a href="#gem5-simulate-limit-reached">gem5 simulate() limit reached</a>.</p>
</div>
<div class="paragraph">
<p>Incorrect behaviour due to: <a href="https://gem5.atlassian.net/browse/GEM5-537" class="bare">https://gem5.atlassian.net/browse/GEM5-537</a>: Exits successfully.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>SEV is not the only thing that can wake up a WFE, it is only an explicit software way to do it.</p>
</div>
<div class="paragraph">
<p>Notably, global monitor operations on memory accesses of regions marked by <a href="#arm-ldxr-and-stxr-instructions">LDAXR and STLXR instructions</a> can also wake up a WFE sleeping core.</p>
</div>
<div class="paragraph">
<p>This is done to allow spinlocks opens to automatically wake up WFE sleeping cores at free time without the need for a explicit SEV.</p>
</div>
<div class="paragraph">
<p>In the shown in the <code>wfe_ldxr_stxr.cpp</code> example, which can only terminate in gem5 user mode simulation because due to this event.</p>
</div>
<div class="paragraph">
<p>Note that that program still terminates when running on top of the Linux kernel as explained at: <a href="#wfe-from-userland">WFE from userland</a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="wfe-from-userland"><a class="anchor" href="#wfe-from-userland"></a><a class="link" href="#wfe-from-userland">33.10.3.1.2. WFE from userland</a></h6>
<div class="paragraph">
<p>WFE and SEV are usable from userland, and are part of an efficient spinlock implementation (which userland should arguably stay away from and rather use the <a href="#futex-system-call">futex system call</a> which allow for non busy sleep instead), which maybe is not something that userland should ever tho and just stick to mutexes?</p>
</div>
<div class="paragraph">
<p>There is a control bit <code>SCTLR_EL1.nTWE</code> that determines if WFE is trapped or not, i.e.: is that bit is set, then it is trapped and EL0 execution raises an exception in EL1.</p>
</div>
<div class="paragraph">
<p>Linux v5.2.1 does not set <code>SCTLR_EL1.nTWE</code> however, tested with <a href="#gem5-tracing">gem5 tracing</a> with <code>--trace ExecAll,Failts</code> and the <a href="#dump-regs">dump_regs kernel module</a> in a full system simulation.</p>
</div>
<div class="paragraph">
<p>The kernel seems to setup nTWE at:</p>
</div>
<div class="paragraph">
<p>include/asm/sysreg.h</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#define SCTLR_EL1_SET	(SCTLR_ELx_M    | SCTLR_ELx_C    | SCTLR_ELx_SA   |\
             ...
			 SCTLR_EL1_NTWE | SCTLR_ELx_IESB | SCTLR_EL1_SPAN |\</pre>
</div>
</div>
<div class="paragraph">
<p>and:</p>
</div>
<div class="paragraph">
<p>mm/proc.S</p>
</div>
<div class="literalblock">
<div class="content">
<pre>	/*
	 * Prepare SCTLR
	 */
	mov_q	x0, SCTLR_EL1_SET</pre>
</div>
</div>
<div class="paragraph">
<p>To reduce the number of instructions from our trace, first we boot, and then we restore a checkpoint after boot with <a href="#gem5-restore-new-script">gem5 checkpoint restore and run a different script</a> with a restore command that runs <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/freestanding/linux/wfe_wfe.S">userland/arch/aarch64/freestanding/linux/wfe_wfe.S</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --gem5-worktree master --gem5-restore 1 --gem5-readfile 'arch/aarch64/freestanding/linux/wfe_wfe.out' --trace ExecAll,Faults,FmtFlag,Thread</pre>
</div>
</div>
<div class="paragraph">
<p>On the traces, we search for <code>wfe</code>, and there are just two hits, so they must be our instructions!</p>
</div>
<div class="paragraph">
<p>The traces then look like this at LKMC 777b7cbbd1d553baf2be9bc2075102be740054dd:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>112285501668497000: Thread: system.cpu: suspend contextId 0
112285501668497000: ExecEnable: system.cpu: A0 T0 : 0x400078    :   wfe                      : IntAlu :  D=0x0000000000000000  flags=(IsSerializeAfter|IsNonSpeculative|IsQuiesce|IsUnverifiable)
112285501668497501: Thread: system.cpu: activate contextId 0
112285501668498000: Thread: system.cpu: suspend contextId 0
112285501668498000: ExecEnable: system.cpu: A0 T0 : 0x40007c    :   wfe                      : IntAlu :  D=0x0000000000000000  flags=(IsSerializeAfter|IsNonSpeculative|IsQuiesce|IsUnverifiable)
112285501909320284: Thread: system.cpu: activate contextId 0
112285501909320500: Faults: IRQ: Invoking Fault (AArch64 target EL):IRQ cpsr:0x4003c5 PC:0x400080 elr:0x400080 newVec: 0xffffff8010082480
112285501909320500: ExecEnable: system.cpu: A0 T0 : @vectors+1152    :   nop                      : IntAlu :   flags=(IsNop)
112285501909321000: ExecEnable: system.cpu: A0 T0 : @vectors+1156    :   nop                      : IntAlu :   flags=(IsNop)

[more exception handler, no ERET here]

112285501923080500: ExecEnable: system.cpu: A0 T0 : @finish_ret_to_user+188    :   ldr   x30, [sp, #240]    : MemRead :  D=0x0000000000000000 A=0xffffff8010cb3fb0  flags=(IsInteger|IsMemRef|IsLoad)
112285501923081000: ExecEnable: system.cpu: A0 T0 : @finish_ret_to_user+192    :   add   sp, sp, #320       : IntAlu :  D=0xffffff8010cb4000  flags=(IsInteger)
112285501923081500: ExecEnable: system.cpu: A0 T0 : 0xffffff8010084144    :   eret                     : IntAlu :  D=0x0000000000000001  flags=(IsControl|IsSerializeAfter|IsNonSpeculative|IsSquashAfter)
112285501923082000: ExecEnable: system.cpu: A0 T0 : 0x400080    :   movz   x0, #0, #0        : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
112285501923082500: ExecEnable: system.cpu: A0 T0 : 0x400084    :   movz   x8, #93, #0       : IntAlu :  D=0x000000000000005d  flags=(IsInteger)
112285501923083000: ExecEnable: system.cpu: A0 T0 : 0x400088    :   svc   #0x0               : IntAlu :   flags=(IsSerializeAfter|IsNonSpeculative|IsSyscall)</pre>
</div>
</div>
<div class="paragraph">
<p>so we conclude that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the second WFE made the CPU stop running instructions at time 112285501668498000 and PC 0x40007c</p>
</li>
<li>
<p>the next thing that happened a long time later (112285501909320500, while a following instruction would happen at 112285501668498000 + 1000) was an interrupt, presumably the <a href="#arm-timer">ARM timer</a></p>
</li>
<li>
<p>after a few interrupt handler instructions, the first <a href="#arm-svc-instruction">ERET</a> instruction exits the handler and comes back directly to the instruction after the WFE at PC 0x400080 == 0x40007c + 4</p>
</li>
<li>
<p>the execution of the interrupt handler woke up the core that was in WFE, and it now continues normal execution past the WFE</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Therefore, a WFE in userland is treated much like a busy loop by the Linux kernel: the kernel does not seem to try and explicitly make up room for other processes as would happen on a futex.</p>
</div>
<div class="paragraph">
<p>The following test checks that SEV events don&#8217;t wake up a futexes, running forever in case of success. In <a href="#gem5-syscall-emulation-multithreading">gem5 syscall emulation multithreading</a>, this is crucial to prevent deadlocks:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/inline_asm/futex_sev.cpp">userland/arch/aarch64/inline_asm/futex_sev.cpp</a></p>
</li>
</ul>
</div>
</div>
<div class="sect5">
<h6 id="armv8-spinlock-pattern"><a class="anchor" href="#armv8-spinlock-pattern"></a><a class="link" href="#armv8-spinlock-pattern">33.10.3.1.3. ARMv8 spinlock pattern</a></h6>
<div class="paragraph">
<p><a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka16277.html" class="bare">http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka16277.html</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>       sev
   1:  wfe
   2:  ldaxr  w1, [w0]
       cbnz   w1, %1b
       stxr   w1, w2, [w0]
       cbnz   w1, %2b</pre>
</div>
</div>
<div class="paragraph">
<p>It is the <a href="#arm-ldxr-and-stxr-instructions">STXR</a> from the unlock on another core that automatically wakes up the spinlock afterwards: <a href="https://stackoverflow.com/questions/32276313/how-is-a-spin-lock-woken-up-in-linux-arm64" class="bare">https://stackoverflow.com/questions/32276313/how-is-a-spin-lock-woken-up-in-linux-arm64</a></p>
</div>
</div>
<div class="sect5">
<h6 id="gem5-arm-wfe"><a class="anchor" href="#gem5-arm-wfe"></a><a class="link" href="#gem5-arm-wfe">33.10.3.1.4. gem5 ARM WFE</a></h6>
<div class="paragraph">
<p>gem5 390a74f59934b85d91489f8a563450d8321b602d does not sleep on the first WFE on either syscall emulation or full system, because the code does:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Fault WfeInst::execute(
    ExecContext *xc, Trace::InstRecord *traceData) const
{
[...]
    if (SevMailbox == 1) {
        SevMailbox = 0;
        PseudoInst::quiesceSkip(tc);
    } else if (tc-&gt;getCpuPtr()-&gt;getInterruptController(
                tc-&gt;threadId())-&gt;checkInterrupts(tc)) {
        PseudoInst::quiesceSkip(tc);
    } else {
        fault = trapWFx(tc, cpsr, scr, true);
        if (fault == NoFault) {
            PseudoInst::quiesce(tc);
        } else {
            PseudoInst::quiesceSkip(tc);
        }
    }</pre>
</div>
</div>
<div class="paragraph">
<p>where <a href="https://en.wiktionary.org/wiki/quiescent">"quiesce" means "sleep"</a> for laymen like Ciro, and <code>quiesceSkip</code> means don&#8217;t sleep.</p>
</div>
<div class="paragraph">
<p><code>SevMailbox</code> is read from <code>MISCREG_SEV_MAILBOX</code> which is initialized to <code>1</code> at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ISA::clear()
{
[...]
    miscRegs[MISCREG_SEV_MAILBOX] = 1;</pre>
</div>
</div>
</div>
<div class="sect5">
<h6 id="arm-yield-instruction"><a class="anchor" href="#arm-yield-instruction"></a><a class="link" href="#arm-yield-instruction">33.10.3.1.5. ARM YIELD instruction</a></h6>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/59311066/how-does-the-arm-yield-instruction-inform-other-threads-that-they-could-start-a" class="bare">https://stackoverflow.com/questions/59311066/how-does-the-arm-yield-instruction-inform-other-threads-that-they-could-start-a</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="arm-ldaxr-and-stlxr-instructions"><a class="anchor" href="#arm-ldaxr-and-stlxr-instructions"></a><a class="link" href="#arm-ldaxr-and-stlxr-instructions">33.10.3.2. ARM LDAXR and STLXR instructions</a></h5>
<div class="paragraph">
<p>Can be used to implement atomic variables, see also:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#atomic-cpp">atomic.cpp</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291" class="bare">https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The ARMv7 analogues are LDREX and STREX.</p>
</div>
</div>
<div class="sect4">
<h5 id="arm-psci"><a class="anchor" href="#arm-psci"></a><a class="link" href="#arm-psci">33.10.3.3. ARM PSCI</a></h5>
<div class="paragraph">
<p>In QEMU, CPU 1 starts in a halted state. This can be observed from GDB, where:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info threads</pre>
</div>
</div>
<div class="paragraph">
<p>shows something like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>* 1    Thread 1 (CPU#0 [running]) lkmc_start
  2    Thread 2 (CPU#1 [halted ]) lkmc_start</pre>
</div>
</div>
<div class="paragraph">
<p>To wake up CPU 1 on QEMU, we must use the Power State Coordination Interface (PSCI) which is documented at: <a href="https://developer.arm.com/docs/den0022/latest/arm-power-state-coordination-interface-platform-design-document" class="bare">https://developer.arm.com/docs/den0022/latest/arm-power-state-coordination-interface-platform-design-document</a>.</p>
</div>
<div class="paragraph">
<p>This interface uses HVC calls, and the calling convention is documented at "SMC CALLING CONVENTION" <a href="https://developer.arm.com/docs/den0028/latest" class="bare">https://developer.arm.com/docs/den0028/latest</a>.</p>
</div>
<div class="paragraph">
<p>If we boot the Linux kernel on QEMU and <a href="#get-device-tree-from-a-running-kernel">dump the auto-generated device tree</a>, we observe that it contains the address of the PSCI CPU_ON call:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>        psci {
                method = "hvc";
                compatible = "arm,psci-0.2", "arm,psci";
                cpu_on = &lt;0xc4000003&gt;;
                migrate = &lt;0xc4000005&gt;;
                cpu_suspend = &lt;0xc4000001&gt;;
                cpu_off = &lt;0x84000002&gt;;
        };</pre>
</div>
</div>
<div class="paragraph">
<p>The Linux kernel wakes up the secondary cores in this exact same way at: <a href="https://github.com/torvalds/linux/blob/v4.19/drivers/firmware/psci.c#L122" class="bare">https://github.com/torvalds/linux/blob/v4.19/drivers/firmware/psci.c#L122</a> We first actually got it working here by grepping the kernel and step debugging that call :-)</p>
</div>
<div class="paragraph">
<p>In gem5, CPU 1 starts woken up from the start, so PSCI is not needed. TODO gem5 actually blows up if we try to do the HVC call, understand why.</p>
</div>
<div class="paragraph">
<p>Bibliography: <a href="https://stackoverflow.com/questions/20055754/arm-start-wakeup-bringup-the-other-cpu-cores-aps-and-pass-execution-start-addre/53473447#53473447" class="bare">https://stackoverflow.com/questions/20055754/arm-start-wakeup-bringup-the-other-cpu-cores-aps-and-pass-execution-start-addre/53473447#53473447</a></p>
</div>
</div>
<div class="sect4">
<h5 id="arm-dmb-instruction"><a class="anchor" href="#arm-dmb-instruction"></a><a class="link" href="#arm-dmb-instruction">33.10.3.4. ARM DMB instruction</a></h5>
<div class="paragraph">
<p>TODO: create and study a minimal examples in gem5 where the DMB instruction leads to less cycles: <a href="https://stackoverflow.com/questions/15491751/real-life-use-cases-of-barriers-dsb-dmb-isb-in-arm" class="bare">https://stackoverflow.com/questions/15491751/real-life-use-cases-of-barriers-dsb-dmb-isb-in-arm</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="arm-timer"><a class="anchor" href="#arm-timer"></a><a class="link" href="#arm-timer">33.10.4. ARM timer</a></h4>
<div class="paragraph">
<p>The ARM timer is the simplest way to generate hardware interrupts periodically, and therefore serves as the simples example of <a href="#arm-gic">ARM GIC</a> usage.</p>
</div>
<div class="paragraph">
<p>Working on QEMU: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/arch/aarch64/timer.c">baremetal/arch/aarch64/timer.c</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --baremetal baremetal/arch/aarch64/timer.c</pre>
</div>
</div>
<div class="paragraph">
<p>Output at lkmc d8dae268c0a3e4e361002aca3b382fedd77f2567 + 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cntv_ctl_el0 0x0
cntfrq_el0 0x3B9ACA0
cntv_cval_el0 0x0
cntvct_el0 0x105113
cntvct_el0 0x1080BC
cntvct_el0 0x10A118

IRQ number 0x1B
cntvct_el0 0x14D25B
cntv_cval_el0 0x3CE9CD6

IRQ number 0x1B
cntvct_el0 0x3CF516F
cntv_cval_el0 0x7893217

IRQ number 0x1B
cntvct_el0 0x789B733
cntv_cval_el0 0xB439642</pre>
</div>
</div>
<div class="paragraph">
<p>and new <code>IRQ number</code> section appears every second, when a clock interrupt is raised!</p>
</div>
<div class="paragraph">
<p>TODO make work on gem5. Fails with <a href="#gem5-simulate-limit-reached">gem5 simulate() limit reached</a> at the first WFI done in main, which means that the interrupt is never raised.</p>
</div>
<div class="paragraph">
<p>Once an interrupt is raised, the interrupt itself sets up a new interrupt to happen in one second in the future after <code>cntv_cval_el0</code> is reached by the counter.</p>
</div>
<div class="paragraph">
<p>The timer is part of the aarch64 specification itself and is documented at: <a href="#armarm8-db">ARMv8 architecture reference manual db</a> Chapter D10 "The Generic Timer in AArch64 state". The key registers to keep in mind are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>CNTVCT_EL0</code>: "Counter-timer Virtual Count register". The increasing current counter value.</p>
</li>
<li>
<p><code>CNTFRQ_EL0</code>: "Counter-timer Frequency register". "Indicates the system counter clock frequency, in Hz."</p>
</li>
<li>
<p><code>CNTV_CTL_EL0</code>: "Counter-timer Virtual Timer Control register". This control register is very simple and only has three fields:</p>
<div class="ulist">
<ul>
<li>
<p><code>CNTV_CTL_EL0.ISTATUS</code> bit: set to 1 when the timer condition is met</p>
</li>
<li>
<p><code>CNTV_CTL_EL0.IMASK</code> bit: if 1, the interrupt does not happen when <code>ISTATUS</code> becomes one</p>
</li>
<li>
<p><code>CNTV_CTL_EL0.ENABLE</code> bit: if 0, the counter is turned off, interrupts don&#8217;t happen</p>
</li>
</ul>
</div>
</li>
<li>
<p><code>CNTV_CVAL_EL0</code>: "Counter-timer Virtual Timer CompareValue register". The interrupt happens when <code>CNTVCT_EL0</code> reaches the value in this register.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Due to <a href="#gem5-vs-qemu">QEMU&#8217;s non-determinism</a>, each consecutive run has slightly different output values.</p>
</div>
<div class="paragraph">
<p>From the terminal output, we can see that the initial clock frequency is 0x3B9ACA0 == 62500000 Hz == 62.5MHz. Grepping QEMU source for that string leads us to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/* Scale factor for generic timers, ie number of ns per tick.
 * This gives a 62.5MHz timer.
 */
#define GTIMER_SCALE 16</pre>
</div>
</div>
<div class="paragraph">
<p>which in turn is used to set the initial reset value of the clock:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    { .name = "CNTFRQ_EL0", .state = ARM_CP_STATE_AA64,
      .opc0 = 3, .opc1 = 3, .crn = 14, .crm = 0, .opc2 = 0,
      .access = PL1_RW | PL0_R, .accessfn = gt_cntfrq_access,
      .fieldoffset = offsetof(CPUARMState, cp15.c14_cntfrq),
      .resetvalue = (1000 * 1000 * 1000) / GTIMER_SCALE,</pre>
</div>
</div>
<div class="paragraph">
<p>where <code>(1000 * 1000 * 1000) / 16 == 62500000</code>.</p>
</div>
<div class="paragraph">
<p>Trying to set the frequency on QEMU by writing to the CNTFRQ register does change the value of future reads, but has no effect on the actual clock frequency as commented on the QEMU source code <a href="https://github.com/qemu/qemu/blob/v4.0.0/target/arm/helper.c#L2647" class="bare">https://github.com/qemu/qemu/blob/v4.0.0/target/arm/helper.c#L2647</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>static const ARMCPRegInfo generic_timer_cp_reginfo[] = {
    /* Note that CNTFRQ is purely reads-as-written for the benefit
     * of software; writing it doesn't actually change the timer frequency.
     * Our reset value matches the fixed frequency we implement the timer at.
     */
    { .name = "CNTFRQ", .cp = 15, .crn = 14, .crm = 0, .opc1 = 0, .opc2 = 0,
      .type = ARM_CP_ALIAS,
      .access = PL1_RW | PL0_R, .accessfn = gt_cntfrq_access,
      .fieldoffset = offsetoflow32(CPUARMState, cp15.c14_cntfrq),
    },</pre>
</div>
</div>
<div class="paragraph">
<p>At each interrupt, we increase the compare value <code>CVAL</code> by about 1x the clock frequency 0x3B9ACA0 so that it will fire again in one second, e.g. <code>0x3CE9CD6 - 0x14D25B == 3B9CA7B</code>. The increment is not perfect because the counter keeps ticking even while our register read and print instructions are running inside the interrupt handler!</p>
</div>
<div class="paragraph">
<p>We then observe that the next interrupt happens soon after CNTV_CVAL_EL0 is reached by CNTVCT_EL0:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cntv_cval_el0 0x3CE9CD6

IRQ number 0x1B
cntvct_el0 0x3CF516F</pre>
</div>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/51094092/how-to-make-timer-irq-work-on-qemu-machine-virt-cpu-cortex-a57" class="bare">https://stackoverflow.com/questions/51094092/how-to-make-timer-irq-work-on-qemu-machine-virt-cpu-cortex-a57</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/44198483/arm-timers-and-interrupts" class="bare">https://stackoverflow.com/questions/44198483/arm-timers-and-interrupts</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="arm-gic"><a class="anchor" href="#arm-gic"></a><a class="link" href="#arm-gic">33.10.5. ARM GIC</a></h4>
<div class="paragraph">
<p>Generic Interrupt Controller.</p>
</div>
<div class="paragraph">
<p>Examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#arm-timer">ARM timer</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>ARM publishes both a GIC standard architecture specification, and specific implementations of these specifications.</p>
</div>
<div class="paragraph">
<p>The specification can be found at: <a href="https://developer.arm.com/docs/ihi0069/latest" class="bare">https://developer.arm.com/docs/ihi0069/latest</a></p>
</div>
<div class="paragraph">
<p>As of 2019Q2 the latest version if v4.0, often called GICv4: <a href="https://static.docs.arm.com/ihi0069/e/Q1-IHI0069E_gic_architecture_specification_v3.1_19_01_21.pdf" class="bare">https://static.docs.arm.com/ihi0069/e/Q1-IHI0069E_gic_architecture_specification_v3.1_19_01_21.pdf</a></p>
</div>
<div class="paragraph">
<p>That document clarifies that GICv2 is a legacy specification only:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Version 2.0 (GICv2) is only described in terms of the GICv3 optional support for legacy operation</pre>
</div>
</div>
<div class="paragraph">
<p>The specific models have names of type GIC-600, GIC-500, etc.</p>
</div>
<div class="paragraph">
<p>In QEMU v4.0.0, the GICv3 can be selected with an extra <code>-machine gic_version=3</code> option.</p>
</div>
<div class="paragraph">
<p>In gem5 3126e84db773f64e46b1d02a9a27892bf6612d30, the GIC is determined by selecting the platform as explained at: <a href="#gem5-arm-platforms">gem5 ARM platforms</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-paging"><a class="anchor" href="#arm-paging"></a><a class="link" href="#arm-paging">33.10.6. ARM paging</a></h4>
<div class="paragraph">
<p>TODO create a minimal working aarch64 example analogous to the x86 one at: <a href="https://github.com/cirosantilli/x86-bare-metal-examples/blob/6dc9a73830fc05358d8d66128f740ef9906f7677/paging.S" class="bare">https://github.com/cirosantilli/x86-bare-metal-examples/blob/6dc9a73830fc05358d8d66128f740ef9906f7677/paging.S</a></p>
</div>
<div class="paragraph">
<p>A general introduction to paging with x86 examples can be found at: <a href="https://cirosantilli.com/x86-paging" class="bare">https://cirosantilli.com/x86-paging</a>.</p>
</div>
<div class="paragraph">
<p>Then, this article is amazing: <a href="https://www.starlab.io/blog/deep-dive-mmu-virtualization-with-xen-on-arm" class="bare">https://www.starlab.io/blog/deep-dive-mmu-virtualization-with-xen-on-arm</a></p>
</div>
<div class="paragraph">
<p>ARM paging is documented at <a href="#armarm8-db">ARMv8 architecture reference manual db</a> Chapter D5 and is mostly called VMSAv8 in the ARMv8 manual (Virtual Memory System Architecture).</p>
</div>
<div class="paragraph">
<p>Paging is enabled by the <code>SCTLR_EL1.M</code> bit.</p>
</div>
<div class="paragraph">
<p>The base table address is selected by the register documented at <a href="#armarm8-db">ARMv8 architecture reference manual db</a> D12.2.111 "TTBR0_EL1, Translation Table Base Register 0 (EL1)".</p>
</div>
<div class="paragraph">
<p>There is also a <code>TTBR1_EL1</code> register, which is for the second translation stage to speed up virtualization: <a href="https://en.wikipedia.org/wiki/Second_Level_Address_Translation" class="bare">https://en.wikipedia.org/wiki/Second_Level_Address_Translation</a> and will not be used in this section.</p>
</div>
<div class="paragraph">
<p>The translation types are described at: <a href="#armarm8-db">ARMv8 architecture reference manual db</a> D5.2.4 "Memory translation granule size".</p>
</div>
<div class="paragraph">
<p>From this we can see that the translation scheme uses up to 4 levels (0 to 3) and has possible granule sizes 4KiB, 16KiB and 64KiB.</p>
</div>
<div class="paragraph">
<p>Page table formats are described at <a href="#armarm8-db">ARMv8 architecture reference manual db</a> D5.3.1 "VMSAv8-64 translation table level 0, level 1, and level 2 descriptor formats".</p>
</div>
</div>
<div class="sect3">
<h4 id="arm-baremetal-bibliography"><a class="anchor" href="#arm-baremetal-bibliography"></a><a class="link" href="#arm-baremetal-bibliography">33.10.7. ARM baremetal bibliography</a></h4>
<div class="paragraph">
<p>First, also consider the userland bibliography: <a href="#arm-assembly-bibliography">Section 30.10, &#8220;ARM assembly bibliography&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The most useful ARM baremetal example sets we&#8217;ve seen so far are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/dwelch67/raspberrypi" class="bare">https://github.com/dwelch67/raspberrypi</a> real hardware</p>
</li>
<li>
<p><a href="https://github.com/dwelch67/qemu_arm_samples" class="bare">https://github.com/dwelch67/qemu_arm_samples</a> QEMU <code>-m vexpress</code></p>
</li>
<li>
<p><a href="https://github.com/bztsrc/raspi3-tutorial" class="bare">https://github.com/bztsrc/raspi3-tutorial</a> real hardware + QEMU <code>-m raspi</code></p>
</li>
<li>
<p><a href="https://github.com/LdB-ECM/Raspberry-Pi" class="bare">https://github.com/LdB-ECM/Raspberry-Pi</a> real hardware</p>
</li>
<li>
<p><a href="https://github.com/BrianSidebotham/arm-tutorial-rpi" class="bare">https://github.com/BrianSidebotham/arm-tutorial-rpi</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="nienfengyaoarmv8-bare-metal"><a class="anchor" href="#nienfengyaoarmv8-bare-metal"></a><a class="link" href="#nienfengyaoarmv8-bare-metal">33.10.7.1. NienfengYao/armv8-bare-metal</a></h5>
<div class="paragraph">
<p><a href="https://github.com/NienfengYao/armv8-bare-metal" class="bare">https://github.com/NienfengYao/armv8-bare-metal</a></p>
</div>
<div class="paragraph">
<p>The only QEMU <code>-m virt</code> aarch64 example set that I can find on the web. Awesome.</p>
</div>
<div class="paragraph">
<p>A large part of the code is taken from the awesome educational OS under 2-clause BSD as can be seen from file headers: <a href="https://github.com/takeharukato/sample-tsk-sw/tree/ce7973aa5d46c9eedb58309de43df3b09d4f8d8d/hal/aarch64" class="bare">https://github.com/takeharukato/sample-tsk-sw/tree/ce7973aa5d46c9eedb58309de43df3b09d4f8d8d/hal/aarch64</a> but Nienfeng largely minimized it.</p>
</div>
<div class="paragraph">
<p>I needed the following minor patches: <a href="https://github.com/NienfengYao/armv8-bare-metal/pull/1" class="bare">https://github.com/NienfengYao/armv8-bare-metal/pull/1</a></p>
</div>
<div class="paragraph">
<p>Handles an SVC and setups and handles the timer about once per second.</p>
</div>
<div class="paragraph">
<p>The source claims GICv3, however if I try to add <code>-machine gic_version=3</code> on their command line with our QEMU v4.0.0, then it blows up at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>static void init_gicc(void)
{
    uint32_t pending_irq;

    /* Disable CPU interface */
    *REG_GIC_GICC_CTLR = GICC_CTLR_DISABLE;</pre>
</div>
</div>
<div class="paragraph">
<p>which tries to write to 0x8010000 according to GDB.</p>
</div>
<div class="paragraph">
<p>Without <code>-machine</code>, QEMU&#8217;s DTB clearly states GICv2, so I&#8217;m starting to wonder if Nienfeng just made a mistake there? The QEMU GICv3 DTB contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>reg = &lt;0x0 0x8000000 0x0 0x10000 0x0 0x80a0000 0x0 0xf60000&gt;;</pre>
</div>
</div>
<div class="paragraph">
<p>and the GICv2 one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>reg = &lt;0x0 0x8000000 0x0 0x10000 0x0 0x8010000 0x0 0x10000&gt;;</pre>
</div>
</div>
<div class="paragraph">
<p>which further confirms that the exception is correct: v2 has a register range at 0x8010000 while in v3 it moved to 0x80a0000 and 0x8010000 is empty.</p>
</div>
<div class="paragraph">
<p>The original source does not mention GICv3 anywhere, only <a href="https://github.com/takeharukato/sample-tsk-sw/blob/c7bbc9dce6b14660bcce8d20735f8c6ebb09396b/hal/aarch64/gic-pl390.c">pl390</a>, which is a specific GIC model that predates the GICv2 spec I believe.</p>
</div>
<div class="paragraph">
<p>TODO if I hack <code>#define GIC_GICC_BASE (GIC_BASE + 0xa0000)</code>, then it goes a bit further, but the next loop never ends.</p>
</div>
</div>
<div class="sect4">
<h5 id="tukl-msdgem5-bare-metal"><a class="anchor" href="#tukl-msdgem5-bare-metal"></a><a class="link" href="#tukl-msdgem5-bare-metal">33.10.7.2. tukl-msd/gem5.bare-metal</a></h5>
<div class="paragraph">
<p><a href="https://github.com/tukl-msd/gem5.bare-metal" class="bare">https://github.com/tukl-msd/gem5.bare-metal</a></p>
</div>
<div class="paragraph">
<p>Reiterated at: <a href="https://stackoverflow.com/questions/43682311/uart-communication-in-gem5-with-arm-bare-metal" class="bare">https://stackoverflow.com/questions/43682311/uart-communication-in-gem5-with-arm-bare-metal</a></p>
</div>
<div class="paragraph">
<p>Basic gem5 aarch64 baremetal setup that just works. Does serial IO and timer through GICv2. Usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Build gem5.
git clone https://gem5.googlesource.com/public/gem5
cd gem5
git checkout 60600f09c25255b3c8f72da7fb49100e2682093a
scons --ignore-style -j`nproc` build/ARM/gem5.opt
cd ..

# Build example.
sudo apt-get install gcc-arm-none-eabi
git clone https://github.com/tukl-msd/gem5.bare-metal
cd gem5.bare-metal
git checkout 6ad1069d4299b775b5491e9252739166bfac9bfe
cd Simple
make CROSS_COMPILE_DIR=/usr/bin

# Run example.
../../gem5/default/build/ARM/gem5.opt' \
  ../../gem5/configs/example/fs.py' \
  --bare-metal \
  --disk-image="$(pwd)/../common/fake.iso" \
  --kernel="$(pwd)/main.elf" \
  --machine-type=RealView_PBX \
  --mem-size=256MB \
;</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="how-we-got-some-baremetal-stuff-to-work"><a class="anchor" href="#how-we-got-some-baremetal-stuff-to-work"></a><a class="link" href="#how-we-got-some-baremetal-stuff-to-work">33.11. How we got some baremetal stuff to work</a></h3>
<div class="paragraph">
<p>It is nice when thing just work.</p>
</div>
<div class="paragraph">
<p>But you can also learn a thing or two from how I actually made them work in the first place.</p>
</div>
<div class="sect3">
<h4 id="find-the-uart-address"><a class="anchor" href="#find-the-uart-address"></a><a class="link" href="#find-the-uart-address">33.11.1. Find the UART address</a></h4>
<div class="paragraph">
<p>Enter the QEMU console:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>Ctrl-X C</pre>
</div>
</div>
<div class="paragraph">
<p>Then do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>info mtree</pre>
</div>
</div>
<div class="paragraph">
<p>And look for <code>pl011</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>    0000000009000000-0000000009000fff (prio 0, i/o): pl011</pre>
</div>
</div>
<div class="paragraph">
<p>On gem5, it is easy to find it on the source. We are using the machine <code>RealView_PBX</code>, and a quick grep leads us to: <a href="https://github.com/gem5/gem5/blob/a27ce59a39ec8fa20a3c4e9fa53e9b3db1199e91/src/dev/arm/RealView.py#L615" class="bare">https://github.com/gem5/gem5/blob/a27ce59a39ec8fa20a3c4e9fa53e9b3db1199e91/src/dev/arm/RealView.py#L615</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>class RealViewPBX(RealView):
    uart = Pl011(pio_addr=0x10009000, int_num=44)</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="aarch64-baremetal-neon-setup"><a class="anchor" href="#aarch64-baremetal-neon-setup"></a><a class="link" href="#aarch64-baremetal-neon-setup">33.11.2. aarch64 baremetal NEON setup</a></h4>
<div class="paragraph">
<p>Inside <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/baremetal/lib/aarch64.S">baremetal/lib/aarch64.S</a> there is a chunk of code that enables floating point operations:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mov x1, 0x3 &lt;&lt; 20
msr cpacr_el1, x1
isb</pre>
</div>
</div>
<div class="paragraph">
<p>CPACR_EL1 is documented at <a href="#armarm8">ARMv8 architecture reference manual</a> D10.2.29 "CPACR_EL1, Architectural Feature Access Control Register".</p>
</div>
<div class="paragraph">
<p>Here we touch the CPACR_EL1.FPEN bits to 3, which enable floating point operations:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>11 This control does not cause any instructions to be trapped.</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>We later also added an enable for the CPACR_EL1.ZEN bits, which are needed for <a href="#arm-sve">ARM SVE</a>.</p>
</div>
<div class="paragraph">
<p>Without CPACR_EL1.FPEN, the <code>printf</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf("got: %c\n", c);</pre>
</div>
</div>
<div class="paragraph">
<p>compiled to a:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>str    q0, [sp, #80]</pre>
</div>
</div>
<div class="paragraph">
<p>which uses NEON registers, and goes into an exception loop.</p>
</div>
<div class="paragraph">
<p>It was a bit confusing because there was a previous <code>printf</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>printf("enter a character\n");</pre>
</div>
</div>
<div class="paragraph">
<p>which did not blow up because GCC compiles it into <code>puts</code> directly since it has no arguments, and that does not generate NEON instructions.</p>
</div>
<div class="paragraph">
<p>The last instructions ran was found with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>while(1)
stepi
end</pre>
</div>
</div>
<div class="paragraph">
<p>or by hacking the QEMU CLI to contain:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-D log.log -d in_asm</pre>
</div>
</div>
<div class="paragraph">
<p>I could not find any previous NEON instruction executed so this led me to suspect that some NEON initialization was required:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://infocenter.arm.com/help/topic/com.arm.doc.dai0527a/DAI0527A_baremetal_boot_code_for_ARMv8_A_processors.pdf" class="bare">http://infocenter.arm.com/help/topic/com.arm.doc.dai0527a/DAI0527A_baremetal_boot_code_for_ARMv8_A_processors.pdf</a> "Bare-metal Boot Code for ARMv8-A Processors"</p>
</li>
<li>
<p><a href="https://community.arm.com/processors/f/discussions/5409/how-to-enable-neon-in-cortex-a8" class="bare">https://community.arm.com/processors/f/discussions/5409/how-to-enable-neon-in-cortex-a8</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/19231197/enable-neon-on-arm-cortex-a-series" class="bare">https://stackoverflow.com/questions/19231197/enable-neon-on-arm-cortex-a-series</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We then tried to copy the code from the "Bare-metal Boot Code for ARMv8-A Processors" document:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>// Disable trapping of accessing in EL3 and EL2.
MSR CPTR_EL3, XZR
MSR CPTR_EL3, XZR
// Disable access trapping in EL1 and EL0.
MOV X1, #(0x3 &lt;&lt; 20) // FPEN disables trapping to EL1.
MSR CPACR_EL1, X1
ISB</pre>
</div>
</div>
<div class="paragraph">
<p>but it entered an exception loop at <code>MSR CPTR_EL3, XZR</code>.</p>
</div>
<div class="paragraph">
<p>We then found out that QEMU <a href="#arm-exception-levels">starts in EL1</a>, and so we kept just the EL1 part, and it worked. Related:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up" class="bare">https://stackoverflow.com/questions/42824706/qemu-system-aarch64-entering-el1-when-emulating-a53-power-up</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/37299524/neon-support-in-armv8-system-mode-qemu" class="bare">https://stackoverflow.com/questions/37299524/neon-support-in-armv8-system-mode-qemu</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="baremetal-tests"><a class="anchor" href="#baremetal-tests"></a><a class="link" href="#baremetal-tests">33.12. Baremetal tests</a></h3>
<div class="paragraph">
<p>Baremetal tests work exactly like <a href="#user-mode-tests">User mode tests</a>, except that you have to add the <code>--mode baremetal</code> option, for example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables --mode baremetal --arch aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>In baremetal, we detect if tests failed by parsing logs for the <a href="#magic-failure-string">Magic failure string</a>.</p>
</div>
<div class="paragraph">
<p>See: <a href="#test-this-repo">Section 38.16, &#8220;Test this repo&#8221;</a> for more useful testing tips.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="android"><a class="anchor" href="#android"></a><a class="link" href="#android">34. Android</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Remember: Android AOSP is a huge undocumented piece of bloatware. It&#8217;s integration into this repo will likely never be super good. See also: <a href="https://cirosantilli.com#android" class="bare">https://cirosantilli.com#android</a></p>
</div>
<div class="paragraph">
<p>Verbose setup description: <a href="https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014" class="bare">https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014</a></p>
</div>
<div class="paragraph">
<p>Download, build and run with the prebuilt AOSP QEMU emulator and the AOSP kernel:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-android \
  --android-base-dir /path/to/your/hd \
  --android-version 8.1.0_r60 \
  download \
  build \
;
./run-android \
  --android-base-dir /path/to/your/hd \
  --android-version 8.1.0_r60 \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-android">build-android</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run-android">run-android</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO how to hack the AOSP kernel, userland and emulator?</p>
</div>
<div class="paragraph">
<p>Other archs work as well as usual with <code>--arch</code> parameter. However, running in non-x86 is very slow due to the lack of KVM.</p>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
<div class="sect2">
<h3 id="android-image-structure"><a class="anchor" href="#android-image-structure"></a><a class="link" href="#android-image-structure">34.1. Android image structure</a></h3>
<div class="paragraph">
<p><a href="https://source.android.com/devices/bootloader/partitions-images" class="bare">https://source.android.com/devices/bootloader/partitions-images</a></p>
</div>
<div class="paragraph">
<p>The messy AOSP generates a ton of images instead of just one.</p>
</div>
<div class="paragraph">
<p>When the emulator launches, we can see them through QEMU <code>-drive</code> arguments:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>emulator: argv[21] = "-initrd"
emulator: argv[22] = "/data/aosp/8.1.0_r60/out/target/product/generic_x86_64/ramdisk.img"
emulator: argv[23] = "-drive"
emulator: argv[24] = "if=none,index=0,id=system,file=/path/to/aosp/8.1.0_r60/out/target/product/generic_x86_64/system-qemu.img,read-only"
emulator: argv[25] = "-device"
emulator: argv[26] = "virtio-blk-pci,drive=system,iothread=disk-iothread,modern-pio-notify"
emulator: argv[27] = "-drive"
emulator: argv[28] = "if=none,index=1,id=cache,file=/path/to/aosp/8.1.0_r60/out/target/product/generic_x86_64/cache.img.qcow2,overlap-check=none,cache=unsafe,l2-cache-size=1048576"
emulator: argv[29] = "-device"
emulator: argv[30] = "virtio-blk-pci,drive=cache,iothread=disk-iothread,modern-pio-notify"
emulator: argv[31] = "-drive"
emulator: argv[32] = "if=none,index=2,id=userdata,file=/path/to/aosp/8.1.0_r60/out/target/product/generic_x86_64/userdata-qemu.img.qcow2,overlap-check=none,cache=unsafe,l2-cache-size=1048576"
emulator: argv[33] = "-device"
emulator: argv[34] = "virtio-blk-pci,drive=userdata,iothread=disk-iothread,modern-pio-notify"
emulator: argv[35] = "-drive"
emulator: argv[36] = "if=none,index=3,id=encrypt,file=/path/to/aosp/8.1.0_r60/out/target/product/generic_x86_64/encryptionkey.img.qcow2,overlap-check=none,cache=unsafe,l2-cache-size=1048576"
emulator: argv[37] = "-device"
emulator: argv[38] = "virtio-blk-pci,drive=encrypt,iothread=disk-iothread,modern-pio-notify"
emulator: argv[39] = "-drive"
emulator: argv[40] = "if=none,index=4,id=vendor,file=/path/to/aosp/8.1.0_r60/out/target/product/generic_x86_64/vendor-qemu.img,read-only"
emulator: argv[41] = "-device"
emulator: argv[42] = "virtio-blk-pci,drive=vendor,iothread=disk-iothread,modern-pio-notify"</pre>
</div>
</div>
<div class="paragraph">
<p>The root directory is the <a href="#initrd">initrd</a> given on the QEMU CLI, which <code>/proc/mounts</code> reports at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rootfs on / type rootfs (ro,seclabel,size=886392k,nr_inodes=221598)</pre>
</div>
</div>
<div class="paragraph">
<p>This contains the <a href="#android-init">Android init</a>, which through <code>.rc</code> must be mounting mounts the drives int o the right places TODO find exact point.</p>
</div>
<div class="paragraph">
<p>The drive order is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>system
cache
userdata
encryptionkey
vendor-qemu</pre>
</div>
</div>
<div class="paragraph">
<p>Then, on the terminal:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mount | grep vd</pre>
</div>
</div>
<div class="paragraph">
<p>gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dev/block/vda1 on /system type ext4 (ro,seclabel,relatime,data=ordered)
/dev/block/vde1 on /vendor type ext4 (ro,seclabel,relatime,data=ordered)
/dev/block/vdb on /cache type ext4 (rw,seclabel,nosuid,nodev,noatime,errors=panic,data=ordered)</pre>
</div>
</div>
<div class="paragraph">
<p>and we see that the order of <code>vda</code>, <code>vdb</code>, etc. matches that in which <code>-drive</code> were given to QEMU.</p>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
<div class="sect3">
<h4 id="android-images-read-only"><a class="anchor" href="#android-images-read-only"></a><a class="link" href="#android-images-read-only">34.1.1. Android images read-only</a></h4>
<div class="paragraph">
<p>From <code>mount</code>, we can see that some of the mounted images are <code>ro</code>.</p>
</div>
<div class="paragraph">
<p>Basically, every image that was given to QEMU as qcow2 is writable, and that qcow2 is an overlay over the actual original image.</p>
</div>
<div class="paragraph">
<p>In order to make <code>/system</code> and <code>/vendor</code> writable by using qcow2 for them as well, we must use the <code>-writable-system</code> option:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-android -- -writable-system</pre>
</div>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://android.stackexchange.com/questions/110927/how-to-mount-system-rewritable-or-read-only-rw-ro/207200#207200" class="bare">https://android.stackexchange.com/questions/110927/how-to-mount-system-rewritable-or-read-only-rw-ro/207200#207200</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/13089694/adb-remount-permission-denied-but-able-to-access-super-user-in-shell-android/43163693#43163693" class="bare">https://stackoverflow.com/questions/13089694/adb-remount-permission-denied-but-able-to-access-super-user-in-shell-android/43163693#43163693</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>then:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>su
mount -o rw,remount /system
date &gt;/system/a</pre>
</div>
</div>
<div class="paragraph">
<p>Now reboot, and relaunch with <code>-writable-system</code> once again to pick up the modified qcow2 images:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-android -- -writable-system</pre>
</div>
</div>
<div class="paragraph">
<p>and the newly created file is still there:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>date &gt;/system/a</pre>
</div>
</div>
<div class="paragraph">
<p><code>/system</code> and <code>/vendor</code> can be nuked quickly with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-android --extra-args snod
./build-android --extra-args vnod</pre>
</div>
</div>
<div class="paragraph">
<p>as mentioned at: <a href="https://stackoverflow.com/questions/29023406/how-to-just-build-android-system-image" class="bare">https://stackoverflow.com/questions/29023406/how-to-just-build-android-system-image</a> and on:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-android --extra-args help</pre>
</div>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
</div>
<div class="sect3">
<h4 id="android-data-partition"><a class="anchor" href="#android-data-partition"></a><a class="link" href="#android-data-partition">34.1.2. Android /data partition</a></h4>
<div class="paragraph">
<p>When I install an app like F-Droid, it goes under <code>/data</code> according to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>find / -iname '*fdroid*'</pre>
</div>
</div>
<div class="paragraph">
<p>and it <a href="#disk-persistency">persists across boots</a>.</p>
</div>
<div class="paragraph">
<p><code>/data</code> is behind a RW LVM device:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/dev/block/dm-0 on /data type ext4 (rw,seclabel,nosuid,nodev,noatime,errors=panic,data=ordered)</pre>
</div>
</div>
<div class="paragraph">
<p>but TODO I can&#8217;t find where it comes from since I don&#8217;t have the CLI tools mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/131519/what-is-this-dm-0-device" class="bare">https://superuser.com/questions/131519/what-is-this-dm-0-device</a></p>
</li>
<li>
<p><a href="https://unix.stackexchange.com/questions/185057/where-does-lvm-store-its-configuration" class="bare">https://unix.stackexchange.com/questions/185057/where-does-lvm-store-its-configuration</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>However, by looking at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-android -- -help</pre>
</div>
</div>
<div class="paragraph">
<p>we see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>-data &lt;file&gt;                   data image (default &lt;datadir&gt;/userdata-qemu.img</pre>
</div>
</div>
<div class="paragraph">
<p>which confirms the suspicion that this data goes in <code>userdata-qemu.img</code>.</p>
</div>
<div class="paragraph">
<p>To reset images to their original state, just remove the qcow2 overlay and regenerate it: <a href="https://stackoverflow.com/questions/54446680/how-to-reset-the-userdata-image-when-building-android-aosp-and-running-it-on-the" class="bare">https://stackoverflow.com/questions/54446680/how-to-reset-the-userdata-image-when-building-android-aosp-and-running-it-on-the</a></p>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="install-android-apps"><a class="anchor" href="#install-android-apps"></a><a class="link" href="#install-android-apps">34.2. Install Android apps</a></h3>
<div class="paragraph">
<p>I don&#8217;t know how to download files from the web on Vanilla android, the default browser does not download anything, and there is no <code>wget</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://android.stackexchange.com/questions/6984/how-to-download-files-from-the-web-in-the-android-browser" class="bare">https://android.stackexchange.com/questions/6984/how-to-download-files-from-the-web-in-the-android-browser</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/26775079/wget-in-android-terminal" class="bare">https://stackoverflow.com/questions/26775079/wget-in-android-terminal</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Installing with <code>adb install</code> does however work: <a href="https://stackoverflow.com/questions/7076240/install-an-apk-file-from-command-prompt" class="bare">https://stackoverflow.com/questions/7076240/install-an-apk-file-from-command-prompt</a></p>
</div>
<div class="paragraph">
<p><a href="https://f-droid.org">F-Droid</a> installed fine like that, however it does not have permission to install apps: <a href="https://www.maketecheasier.com/install-apps-from-unknown-sources-android/" class="bare">https://www.maketecheasier.com/install-apps-from-unknown-sources-android/</a></p>
</div>
<div class="paragraph">
<p>And the <code>Settings</code> app crashes so I can&#8217;t change it, logcat contains:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>No service published for: wifip2p</pre>
</div>
</div>
<div class="paragraph">
<p>which is mentioned at: <a href="https://stackoverflow.com/questions/47839955/android-8-settings-app-crashes-on-emulator-with-clean-aosp-build" class="bare">https://stackoverflow.com/questions/47839955/android-8-settings-app-crashes-on-emulator-with-clean-aosp-build</a></p>
</div>
<div class="paragraph">
<p>We also tried to enable it from the command line with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>settings put secure install_non_market_apps 1</pre>
</div>
</div>
<div class="paragraph">
<p>as mentioned at: <a href="https://android.stackexchange.com/questions/77280/allow-unknown-sources-from-terminal-without-going-to-settings-app" class="bare">https://android.stackexchange.com/questions/77280/allow-unknown-sources-from-terminal-without-going-to-settings-app</a> but it didn&#8217;t work either.</p>
</div>
<div class="paragraph">
<p>No person alive seems to know how to pre-install apps on AOSP: <a href="https://stackoverflow.com/questions/6249458/pre-installing-android-application" class="bare">https://stackoverflow.com/questions/6249458/pre-installing-android-application</a></p>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
</div>
<div class="sect2">
<h3 id="android-init"><a class="anchor" href="#android-init"></a><a class="link" href="#android-init">34.3. Android init</a></h3>
<div class="paragraph">
<p>For Linux in general, see: <a href="#init">Section 7, &#8220;init&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>The <code>/init</code> executable interprets the <code>/init.rc</code> files, which is in a custom Android init system language: <a href="https://android.googlesource.com/platform/system/core/+/ee0e63f71d90537bb0570e77aa8a699cc222cfaf/init/README.md" class="bare">https://android.googlesource.com/platform/system/core/+/ee0e63f71d90537bb0570e77aa8a699cc222cfaf/init/README.md</a></p>
</div>
<div class="paragraph">
<p>The top of that file then sources other <code>.rc</code> files present on the root directory:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>import /init.environ.rc
import /init.usb.rc
import /init.${ro.hardware}.rc
import /vendor/etc/init/hw/init.${ro.hardware}.rc
import /init.usb.configfs.rc
import /init.${ro.zygote}.rc</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: how is <code>ro.hardware</code> determined? <a href="https://stackoverflow.com/questions/20572781/android-boot-where-is-the-init-hardware-rc-read-in-init-c-where-are-servic" class="bare">https://stackoverflow.com/questions/20572781/android-boot-where-is-the-init-hardware-rc-read-in-init-c-where-are-servic</a> It is a system property and can be obtained with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>getprop ro.hardware</pre>
</div>
</div>
<div class="paragraph">
<p>This gives:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ranchu</pre>
</div>
</div>
<div class="paragraph">
<p>which is the codename for the QEMU virtual platform we are running on: <a href="https://www.oreilly.com/library/view/android-system-programming/9781787125360/9736a97c-cd09-40c3-b14d-955717648302.xhtml" class="bare">https://www.oreilly.com/library/view/android-system-programming/9781787125360/9736a97c-cd09-40c3-b14d-955717648302.xhtml</a></p>
</div>
<div class="paragraph">
<p>TODO: is it possible to add a custom <code>.rc</code> file without modifying the initrd that <a href="#android-image-structure">gets mounted on root</a>? <a href="https://stackoverflow.com/questions/9768103/make-persistent-changes-to-init-rc" class="bare">https://stackoverflow.com/questions/9768103/make-persistent-changes-to-init-rc</a></p>
</div>
<div class="paragraph">
<p>Tested on: <code>8.1.0_r60</code>.</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="benchmark-this-repo"><a class="anchor" href="#benchmark-this-repo"></a><a class="link" href="#benchmark-this-repo">35. Benchmark this repo</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>TODO: didn&#8217;t fully port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.</p>
</div>
<div class="paragraph">
<p>In this section document how benchmark builds and runs of this repo, and how to investigate what the bottleneck is.</p>
</div>
<div class="paragraph">
<p>Ideally, we should setup an automated build server that benchmarks those things continuously for us, but our <a href="#travis">Travis</a> attempt failed.</p>
</div>
<div class="paragraph">
<p>So currently, we are running benchmarks manually when it seems reasonable and uploading them to: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat-regression" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat-regression</a></p>
</div>
<div class="paragraph">
<p>All benchmarks were run on the <a href="#p51">2017 Lenovo ThinkPad P51</a> machine, unless stated otherwise.</p>
</div>
<div class="paragraph">
<p>Run all benchmarks and upload the results:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd ..
git clone https://github.com/cirosantilli/linux-kernel-module-cheat-regression
cd -
./bench-all -A</pre>
</div>
</div>
<div class="sect2">
<h3 id="continuous-integration"><a class="anchor" href="#continuous-integration"></a><a class="link" href="#continuous-integration">35.1. Continuous integration</a></h3>
<div class="paragraph">
<p>We have explored a few Continuous integration solutions.</p>
</div>
<div class="paragraph">
<p>We haven&#8217;t setup any of them yet.</p>
</div>
<div class="sect3">
<h4 id="travis"><a class="anchor" href="#travis"></a><a class="link" href="#travis">35.1.1. Travis</a></h4>
<div class="paragraph">
<p>We tried to automate it on Travis with <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/.travis.yml">.travis.yml</a> but it hits the current 50 minute job timeout: <a href="https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523" class="bare">https://travis-ci.org/cirosantilli/linux-kernel-module-cheat/builds/296454523</a> And I bet it would likely hit a disk maxout either way if it went on.</p>
</div>
</div>
<div class="sect3">
<h4 id="circleci"><a class="anchor" href="#circleci"></a><a class="link" href="#circleci">35.1.2. CircleCI</a></h4>
<div class="paragraph">
<p>This setup successfully built gem5 on every commit: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/.circleci/config.yml">.circleci/config.yml</a></p>
</div>
<div class="paragraph">
<p>Enabling it is however blocked on: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79</a> so we disabled the builds on the web UI.</p>
</div>
<div class="paragraph">
<p>If that ever gets done, we will also need to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>convert this to a nightly with a workflow, to save server resources: <a href="https://circleci.com/docs/2.0/configuration-reference/#triggers" class="bare">https://circleci.com/docs/2.0/configuration-reference/#triggers</a></p>
</li>
<li>
<p>download the prebuilt disk images and enable caches to save the images across runs</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>A build took about 1 hour of a core, and the free tier allows for 1000 minutes per month: <a href="https://circleci.com/pricing/" class="bare">https://circleci.com/pricing/</a> so about 17 hours. The cheapest non-free setup seems to be 50 dollars per month gets us infinite build minutes per month and 2 containers, so we could scale things to run in under 24 hours.</p>
</div>
<div class="paragraph">
<p>There is no result reporting web UI however&#8230;&#8203; but neither does GitLab CI: <a href="https://gitlab.com/gitlab-org/gitlab-ce/issues/17081" class="bare">https://gitlab.com/gitlab-org/gitlab-ce/issues/17081</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="benchmark-this-repo-benchmarks"><a class="anchor" href="#benchmark-this-repo-benchmarks"></a><a class="link" href="#benchmark-this-repo-benchmarks">35.2. Benchmark this repo benchmarks</a></h3>
<div class="sect3">
<h4 id="benchmark-linux-kernel-boot"><a class="anchor" href="#benchmark-linux-kernel-boot"></a><a class="link" href="#benchmark-linux-kernel-boot">35.2.1. Benchmark Linux kernel boot</a></h4>
<div class="paragraph">
<p>Run all kernel boot benchmarks for one arch:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-test-boot --size 3 &amp;&amp; ./test-boot --all-archs --all-emulators --size 3
cat "$(./getvar test_boot_benchmark_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>Sample results at LKMC 8fb9db39316d43a6dbd571e04dd46ae73915027f:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cmd ./run --arch x86_64 --eval './linux/poweroff.out'
time 8.25
exit_status 0

cmd ./run --arch x86_64 --eval './linux/poweroff.out' --kvm
time 1.22
exit_status 0

cmd ./run --arch x86_64 --eval './linux/poweroff.out' --trace exec_tb
time 8.83
exit_status 0
instructions 2244297

cmd ./run --arch x86_64 --eval 'm5 exit' --emulator gem5
time 213.39
exit_status 0
instructions 318486337

cmd ./run --arch arm --eval './linux/poweroff.out'
time 6.62
exit_status 0

cmd ./run --arch arm --eval './linux/poweroff.out' --trace exec_tb
time 6.90
exit_status 0
instructions 776374

cmd ./run --arch arm --eval 'm5 exit' --emulator gem5
time 118.46
exit_status 0
instructions 153023392

cmd ./run --arch arm --eval 'm5 exit' --emulator gem5 -- --cpu-type=HPI --caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB
time 2250.40
exit_status 0
instructions 151981914

cmd ./run --arch aarch64 --eval './linux/poweroff.out'
time 4.94
exit_status 0

cmd ./run --arch aarch64 --eval './linux/poweroff.out' --trace exec_tb
time 5.04
exit_status 0
instructions 233162

cmd ./run --arch aarch64 --eval 'm5 exit' --emulator gem5
time 70.89
exit_status 0
instructions 124346081

cmd ./run --arch aarch64 --eval 'm5 exit' --emulator gem5 -- --cpu-type=HPI --caches --l2cache --l1d_size=1024kB --l1i_size=1024kB --l2_size=1024kB --l3_size=1024kB
time 381.86
exit_status 0
instructions 124564620

cmd ./run --arch aarch64 --eval 'm5 exit' --emulator gem5 --gem5-build-type fast
time 58.00
exit_status 0
instructions 124346081

cmd ./run --arch aarch64 --eval 'm5 exit' --emulator gem5 --gem5-build-type debug
time 1022.03
exit_status 0
instructions 124346081</pre>
</div>
</div>
<div class="paragraph">
<p>TODO: aarch64 gem5 and QEMU use the same kernel, so why is the gem5 instruction count so much much higher?</p>
</div>
<div class="paragraph">
<p><a href="#p51">2017 Lenovo ThinkPad P51</a> Ubuntu 19.10 LKMC b11e3cd9fb5df0e3fe61de28e8264bbc95ea9005 gem5 e779c19dbb51ad2f7699bd58a5c7827708e12b55 aarch64: 143s. Why huge increases from 70s on above table? Kernel size is also huge BTW: 147MB.</p>
</div>
<div class="paragraph">
<p>Note that <a href="https://gem5.atlassian.net/browse/GEM5-337" class="bare">https://gem5.atlassian.net/browse/GEM5-337</a> "ARM PAuth patch slows down Linux boot 2x from 2 minutes to 4 minutes" was already semi fixed at that point.</p>
</div>
<div class="paragraph">
<p>Same but with <a href="#buildroot-vanilla-kernel">Buildroot vanilla kernel</a> (kernel v4.19): 44s to blow up at "Please append a correct "root=" boot option; here are the available partitions" because missing some filesystem mount option. But likely wouldn&#8217;t be much more until after boot since we are almost already done by then! Therefore this vanilla kernel is much much faster! TODO find which config or kernel commit added so much time! Also that kernel is tiny at 8.5MB.</p>
</div>
<div class="paragraph">
<p>Same but hacking <code>BR2_LINUX_KERNEL_LATEST_VERSION=y</code> and <code>BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_3=y</code> which reaches kernel 5.3.14 which closer to the LKMC one 5.4.3: 40s, which is very similar for the older kernel. Therefore it does not loook like it is a problem of kernel code changes, but rather of configs.</p>
</div>
<div class="paragraph">
<p>Same but with: <a href="#gem5-arm-linux-kernel-patches">gem5 arm Linux kernel patches</a> at v4.15: 73s, kernel size: 132M.</p>
</div>
<div class="paragraph">
<p>On Ubuntu 20.04, LKMC d3f8d3e99f2e554aae6c3b325b350bcf7f3f087f (Linux kernel 5.4.3), gem5 6bc2111c9674d0c8db22f6a6adcc00e49625aabd (sept 2020):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --quit-after-boot</pre>
</div>
</div>
<div class="paragraph">
<p>took 193s. With some minimal newer kernel boot patches:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>kernel v5.7: 238s</p>
</li>
<li>
<p>kernel v5.8: 239s</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>On Ubuntu 20.04 gem5 3ca404da175a66e0b958165ad75eb5f54cb5e772 this took 22 minutes 53 seconds:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run -aa -eg --cpus 2 --tmux --quit-after-boot -- --cpu-type DerivO3CPU --caches</pre>
</div>
</div>
<div class="sect4">
<h5 id="gem5-arm-hpi-boot-takes-much-longer-than-aarch64"><a class="anchor" href="#gem5-arm-hpi-boot-takes-much-longer-than-aarch64"></a><a class="link" href="#gem5-arm-hpi-boot-takes-much-longer-than-aarch64">35.2.1.1. gem5 arm HPI boot takes much longer than aarch64</a></h5>
<div class="paragraph">
<p>TODO 62f6870e4e0b384c4bd2d514116247e81b241251 takes 33 minutes to finish at 62f6870e4e0b384c4bd2d514116247e81b241251:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cmd ./run --arch arm --eval 'm5 exit' --emulator gem5 -- --caches --cpu-type=HPI</pre>
</div>
</div>
<div class="paragraph">
<p>while aarch64 only 7 minutes.</p>
</div>
<div class="paragraph">
<p>I had previously documented on README 10 minutes at: 2eff007f7c3458be240c673c32bb33892a45d3a0 found with <code>git log</code> search for <code>10 minutes</code>. But then I checked out there, run it, and kernel panics before any messages come out. Lol?</p>
</div>
<div class="paragraph">
<p>Logs of the runs can be found at: <a href="https://github.com/cirosantilli2/gem5-issues/tree/0df13e862b50ae20fcd10bae1a9a53e55d01caac/arm-hpi-slow" class="bare">https://github.com/cirosantilli2/gem5-issues/tree/0df13e862b50ae20fcd10bae1a9a53e55d01caac/arm-hpi-slow</a></p>
</div>
<div class="paragraph">
<p>The cycle count is higher for <code>arm</code>, 350M vs 250M for <code>aarch64</code>, not nowhere near the 5x runtime time increase.</p>
</div>
<div class="paragraph">
<p>A quick look at the boot logs show that they are basically identical in structure: the same operations appear more ore less on both, and there isn&#8217;t one specific huge time pit in arm: it is just that every individual operation seems to be taking a lot longer.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-x86-64-derivo3cpu-boot-panics"><a class="anchor" href="#gem5-x86-64-derivo3cpu-boot-panics"></a><a class="link" href="#gem5-x86-64-derivo3cpu-boot-panics">35.2.1.2. gem5 x86_64 DerivO3CPU boot panics</a></h5>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli2/gem5-issues/issues/2" class="bare">https://github.com/cirosantilli2/gem5-issues/issues/2</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>Kernel panic - not syncing: Attempted to kill the idle task!</pre>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="benchmark-emulators-on-userland-executables"><a class="anchor" href="#benchmark-emulators-on-userland-executables"></a><a class="link" href="#benchmark-emulators-on-userland-executables">35.2.2. Benchmark emulators on userland executables</a></h4>
<div class="paragraph">
<p>Let&#8217;s see how fast our simulators are running some well known or easy to understand userland benchmarks!</p>
</div>
<div class="paragraph">
<p>TODO: would be amazing to have an automated guest instructions per second count, but I&#8217;m not sure how to do that nicely for QEMU: <a href="#qemu-get-guest-instruction-count">QEMU get guest instruction count</a>.</p>
</div>
<div class="paragraph">
<p>TODO: automate this further, produce the results table automatically, possibly by generalizing <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-executables">test-executables</a>.</p>
</div>
<div class="paragraph">
<p>For now we can just run on gem5 to estimate the instruction count per input size and extrapolate?</p>
</div>
<div class="paragraph">
<p>For example, the simplest scalable CPU content would be an <a href="#c-busy-loop">C busy loop</a>, so let&#8217;s start by analyzing that one.</p>
</div>
<div class="paragraph">
<p>Summary of manually collected results on <a href="#p51">2017 Lenovo ThinkPad P51</a> at LKMC a18f28e263c91362519ef550150b5c9d75fa3679 + 1: <a href="#table-busy-loop-dmips">Table 7, &#8220;Busy loop MIPS for different simulator setups&#8221;</a>. As expected, the less native/more detailed/more complex simulations are slower!</p>
</div>
<table id="table-busy-loop-dmips" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 7. Busy loop MIPS for different simulator setups</caption>
<colgroup>
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
<col style="width: 10%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Comment</th>
<th class="tableblock halign-left valign-top">LKMC</th>
<th class="tableblock halign-left valign-top">Benchmark build</th>
<th class="tableblock halign-left valign-top">Emulator command</th>
<th class="tableblock halign-left valign-top">Loops</th>
<th class="tableblock halign-left valign-top">Time (s)</th>
<th class="tableblock halign-left valign-top">Instruction count</th>
<th class="tableblock halign-left valign-top">Approximate MIPS</th>
<th class="tableblock halign-left valign-top">Hardware version</th>
<th class="tableblock halign-left valign-top">Host OS</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Native busy loop</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a7ae8e6a8e29ef46d79eb9178d8599d1faeea0e5 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --emulator native --userland userland/gcc/busy_loop.c --cli-args 10000000000</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">27</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="#p51">2017 Lenovo ThinkPad P51</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ubuntu 20.04</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">QEMU aarch64 busy loop</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --userland userland/gcc/busy_loop.c --cli-args 10000000000</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^10</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">68</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.1 * 10^11 (approx)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2000</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --static --userland userland/gcc/busy_loop.c --cli-args 1000000</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">18</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.4005699 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.3</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 empty C program statically linked</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">eb22fd3b6e7fff7e9ef946a88b208debf5b419d5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/empty.c">userland/c/empty.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --static --userland userland/c/empty.c</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">5475</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">872cb227fdc0b4d60acc7840889d567a6936b6e1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ubuntu 20.04</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 empty C program dynamically linked</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">eb22fd3b6e7fff7e9ef946a88b208debf5b419d5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/empty.c">userland/c/empty.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --userland userland/c/empty.c</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">106999</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">872cb227fdc0b4d60acc7840889d567a6936b6e1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ubuntu 20.04</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop for a debug build</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --gem5-build-type debug --static --userland userland/gcc/busy_loop.c --cli-args 100000</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">33</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.405682 * 10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.07</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop for a fast build</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0d5a41a3f88fcd7ed40fc19474fe5aed0463663f + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0 -static</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --gem5-build-type fast --static --userland userland/gcc/busy_loop.c --cli-args 1000000</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">15</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.4005699 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.6</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop for a <a href="#gem5-cpu-types">TimingSimpleCPU</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --arch aarch64 --static --userland userland/gcc/busy_loop.c --cli-args 1000000 -- --cpu-type TimingSimpleCPU --caches</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">26</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.4005699 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.9</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop for a <a href="#gem5-cpu-types">MinorCPU</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --arch aarch64 --userland userland/gcc/busy_loop.c --cli-args 1000000 -- --cpu-type MinorCPU --caches</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">31</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.1018152 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.4</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 busy loop for a <a href="#gem5-cpu-types">DerivO3CPU</a></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --userland userland/gcc/busy_loop.c --userland args 1000000 -- --cpu-type DerivO3CPU --caches</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">52</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.1018128 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.2</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">a18f28e263c91362519ef550150b5c9d75fa3679 + 1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --gem5-build-id MOESI_CMP_directory -- --cpu-type DerivO3CPU --caches --ruby</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1 * 1000000 = 10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">63</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.1005150 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.2</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">glibc C pre-main effects</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/m5ops.c">userland/c/m5ops.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64 --cli-args e</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.26479 * 10^5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.05</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">glibc C pre-main <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/m5ops.c">userland/c/m5ops.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64 --cli-args e --gem5-build-type debug</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1.26479 * 10^5</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.05</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">glibc C++ pre-main <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/m5ops.cpp">userland/cpp/m5ops.cpp</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64 --cli-args e</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.385012 * 10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">glibc C++ pre-main <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/m5ops.cpp">userland/cpp/m5ops.cpp</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64 --cli-args e --gem5-build-type debug</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">25</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.385012 * 10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.1</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">gem5 optimized build immediate exit on first instruction to benchmark the simulator startup time</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">immediate exit <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/gem5_exit.S">userland/freestanding/gem5_exit.S</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">same as above but debug build</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">ab6f7331406b22f8ab6e2df5f8b8e464fb35b611</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/freestanding/gem5_exit.S">userland/freestanding/gem5_exit.S</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>gem5 --arch aarch64 --gem5-build-type debug</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Check the effect of an ExecAll log (log every instruction) on execution time, compare to analogous run without it. <code>trace.txt</code> size: 3.5GB. 5x slowdown observed with output to a hard disk.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">d29a07ddad499f273cc90dd66e40f8474b5dfc40</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a> <code>-O0</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./run --arch aarch64 --emulator gem5 --userland userland/gcc/busy_loop.c --cli-args 1000000 --gem5-worktree master --trace ExecAll</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">10^6</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2.4106774 * 10^7</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">136</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">0.2</p></td>
<td class="tableblock halign-left valign-top"></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Same as above but with run command manually hacked to output to a ramfs. Slightly faster, but the bulk was still just in log format operations!</p></td>
</tr>
</tbody>
</table>
<div class="paragraph">
<p>The first step is to determine a number of loops that will run long enough to have meaningful results, but not too long that we will get bored, so about 1 minute.</p>
</div>
<div class="paragraph">
<p>On our <a href="#p51">2017 Lenovo ThinkPad P51</a> machine, we found 10^7 (10 million == 1000 times 10000) loops to be a good number for a gem5 atomic simulation:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --userland userland/gcc/busy_loop.c --cli-args '1 10000000'
./gem5-stat --arch aarch64 sim_insts</pre>
</div>
</div>
<div class="paragraph">
<p>as it gives:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>time: 00:01:40</p>
</li>
<li>
<p>instructions: 110018162 ~ 110 millions</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>so ~ 110 million instructions / 100 seconds makes ~ 1 MIPS (million instructions per second).</p>
</div>
<div class="paragraph">
<p>This experiment also suggests that each loop is about 11 instructions long (110M instructions / 10M loops), which we confirm at <a href="#c-busy-loop">Section 36.2, &#8220;C busy loop&#8221;</a>, bingo!</p>
</div>
<div class="paragraph">
<p>Then for QEMU, we experimentally turn the number of loops up to 10^10 loops (<code>100000 100000</code>), which contains an expected 11 * 10^10 instructions, and the runtime is 00:01:08, so we have 1.1 * 10^11 instruction / 68 seconds ~ 2 * 10^9 = 2000 MIPS!</p>
</div>
<div class="paragraph">
<p>We can then repeat the experiment for other gem5 CPUs to see how they compare.</p>
</div>
<div class="sect4">
<h5 id="user-mode-vs-full-system-benchmark"><a class="anchor" href="#user-mode-vs-full-system-benchmark"></a><a class="link" href="#user-mode-vs-full-system-benchmark">35.2.2.1. User mode vs full system benchmark</a></h5>
<div class="paragraph">
<p>Let&#8217;s see if user mode runs considerably faster than full system or not, ignoring the kernel boot.</p>
</div>
<div class="paragraph">
<p>First we build dhrystonee manually statically since dynamic linking is broken in gem5 as explained at: <a href="#gem5-syscall-emulation-mode">Section 11.7, &#8220;gem5 syscall emulation mode&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>gem5 user mode:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --arch arm --config 'BR2_PACKAGE_DHRYSTONE=y'
make \
  -B \
  -C "$(./getvar --arch arm buildroot_build_build_dir)/dhrystone-2" \
  CC="$(./run-toolchain --arch arm --print-tool gcc)" \
  CFLAGS=-static \
;
time \
  ./run \
  --arch arm \
  --emulator gem5 \
  --userland "$(./getvar --arch arm buildroot_build_build_dir)/dhrystone-2/dhrystone" \
  --cli-args 'asdf qwer' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>gem5 full system:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>time \
  ./run \
  --arch arm \
  --eval-after './gem5.sh' \
  --emulator gem5
  --gem5-readfile 'dhrystone 100000' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>QEMU user mode:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>time qemu-arm "$(./getvar --arch arm buildroot_build_build_dir)/dhrystone-2/dhrystone" 100000000</pre>
</div>
</div>
<div class="paragraph">
<p>QEMU full system:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>time \
  ./run \
  --arch arm \
  --eval-after 'time dhrystone 100000000;./linux/poweroff.out' \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Result on <a href="#p51">2017 Lenovo ThinkPad P51</a> at bad30f513c46c1b0995d3a10c0d9bc2a33dc4fa0:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>gem5 user: 33 seconds</p>
</li>
<li>
<p>gem5 full system: 51 seconds</p>
</li>
<li>
<p>QEMU user: 45 seconds</p>
</li>
<li>
<p>QEMU full system: 223 seconds</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="benchmark-builds"><a class="anchor" href="#benchmark-builds"></a><a class="link" href="#benchmark-builds">35.2.3. Benchmark builds</a></h4>
<div class="paragraph">
<p>The build times are calculated after doing <code>./configure</code> and <a href="https://buildroot.org/downloads/manual/manual.html#_offline_builds"><code>make source</code></a>, which downloads the sources, and basically benchmarks the <a href="#benchmark-internets">Internet</a>.</p>
</div>
<div class="paragraph">
<p>Sample build time at 2c12b21b304178a81c9912817b782ead0286d282: 28 minutes, 15 with full ccache hits. Breakdown: 19% GCC, 13% Linux kernel, 7% uclibc, 6% host-python, 5% host-qemu, 5% host-gdb, 2% host-binutils</p>
</div>
<div class="paragraph">
<p>Buildroot automatically stores build timestamps as milliseconds since Epoch. Convert to minutes:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>awk -F: 'NR==1{start=$1}; END{print ($1 - start)/(60000.0)}' "$(./getvar buildroot_build_build_dir)/build-time.log"</pre>
</div>
</div>
<div class="paragraph">
<p>Or to conveniently do a clean build without affecting your current one:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./bench-all -b
cat ../linux-kernel-module-cheat-regression/*/build-time.log</pre>
</div>
</div>
<div class="sect4">
<h5 id="find-which-buildroot-packages-are-making-the-build-slow-and-big"><a class="anchor" href="#find-which-buildroot-packages-are-making-the-build-slow-and-big"></a><a class="link" href="#find-which-buildroot-packages-are-making-the-build-slow-and-big">35.2.3.1. Find which Buildroot packages are making the build slow and big</a></h5>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot -- graph-build graph-size graph-depends
cd "$(./getvar buildroot_build_dir)/graphs"
xdg-open build.pie-packages.pdf
xdg-open graph-depends.pdf
xdg-open graph-size.pdf</pre>
</div>
</div>
<div class="sect5">
<h6 id="prebuilt-toolchain"><a class="anchor" href="#prebuilt-toolchain"></a><a class="link" href="#prebuilt-toolchain">35.2.3.1.1. Buildroot use prebuilt host toolchain</a></h6>
<div class="paragraph">
<p>The biggest build time hog is always GCC, and it does not look like we can use a precompiled one: <a href="https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain" class="bare">https://stackoverflow.com/questions/10833672/buildroot-environment-with-host-toolchain</a></p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="benchmark-buildroot-build-baseline"><a class="anchor" href="#benchmark-buildroot-build-baseline"></a><a class="link" href="#benchmark-buildroot-build-baseline">35.2.3.2. Benchmark Buildroot build baseline</a></h5>
<div class="paragraph">
<p>This is the minimal build we could expect to get away with.</p>
</div>
<div class="paragraph">
<p>We will run this whenever the Buildroot submodule is updated.</p>
</div>
<div class="paragraph">
<p>On the upstream Buildroot repo at :</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./bench-all -B</pre>
</div>
</div>
<div class="paragraph">
<p>Sample time on 2017.08: 11 minutes, 7 with full ccache hits. Breakdown: 47% GCC, 15% Linux kernel, 9% uclibc, 5% host-binutils. Conclusions:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>we have bloated our kernel build 3x with all those delicious features :-)</p>
</li>
<li>
<p>GCC time increased 1.5x by our bloat, but its percentage of the total was greatly reduced, due to new packages being introduced.</p>
<div class="paragraph">
<p><code>make graph-depends</code> shows that most new dependencies come from QEMU and GDB, which we can&#8217;t get rid of anyway.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>A quick look at the system monitor reveals that the build switches between times when:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPUs are at a max, memory is fine. So we must be CPU / memory speed bound. I bet that this happens during heavy compilation.</p>
</li>
<li>
<p>CPUs are not at a max, and memory is fine. So we are likely disk bound. I bet that this happens during configuration steps.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is consistent with the fact that ccache reduces the build time only partially, since ccache should only overcome the CPU bound compilation steps, but not the disk bound ones.</p>
</div>
<div class="paragraph">
<p>The instructions counts varied very little between the baseline and LKMC, so runtime overhead is not a big deal apparently.</p>
</div>
<div class="paragraph">
<p>Size:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>bzImage</code>: 4.4M</p>
</li>
<li>
<p><code>rootfs.cpio</code>: 1.6M</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Zipped: 4.9M, <code>rootfs.cpio</code> deflates 50%, <code>bzImage</code> almost nothing.</p>
</div>
</div>
<div class="sect4">
<h5 id="benchmark-gem5-build"><a class="anchor" href="#benchmark-gem5-build"></a><a class="link" href="#benchmark-gem5-build">35.2.3.3. Benchmark gem5 build</a></h5>
<div class="paragraph">
<p>How long it takes to build gem5 itself.</p>
</div>
<div class="paragraph">
<p>We will update this whenever the gem5 submodule is updated.</p>
</div>
<div class="paragraph">
<p>All benchmarks done on <a href="#p51">2017 Lenovo ThinkPad P51</a>.</p>
</div>
<div class="paragraph">
<p>Get results with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./bench-all --emulator gem5
tail -n+1 ../linux-kernel-module-cheat-regression/*/gem5-bench-build-*.txt</pre>
</div>
</div>
<div class="paragraph">
<p>Ubuntu 19.10, GCC 9.2.1, LKMC 7c6bb29bc89ec3f1056c0680c3f08bd64018a7bc, gem5 d7d9bc240615625141cd6feddbadd392457e49eb (2020-02-18), <code>./build --arch aarch64 --gem5-worktree master --no-cache</code>: 19m 33s TODO must investigate why it got so much worse.</p>
</div>
<div class="paragraph">
<p>Ubuntu 20.04, GCC 9.3.0, LKMC 6275f70ed8862d8fe4e58ca4524a6994d254be35, gem5 d9cb548d83fa81858599807f54b52e5be35a6b03 (2020-05-06), <code>./build --arch aarch64 --gem5-worktree master --no-cache</code>: 28m!!! It&#8217;s out of control.</p>
</div>
<div class="paragraph">
<p>Same but gem5 d7d9bc240615625141cd6feddbadd392457e49eb (2018-06-17) hacked with <code>-Wnoerror</code>: 11m 37s. So there was a huge regression in the last two years! We have to find it out.</p>
</div>
<div class="paragraph">
<p>A profiling of the build has been done at: <a href="https://gem5.atlassian.net/browse/GEM5-277" class="bare">https://gem5.atlassian.net/browse/GEM5-277</a> Analysis there showed that d7d9bc240615625141cd6feddbadd392457e49eb (2018-06-17) is also composed of 50% pybind11 and with no obvious time sinks.</p>
</div>
<div class="sect5">
<h6 id="pybind11-accounts-for-50-of-gem5-build-time"><a class="anchor" href="#pybind11-accounts-for-50-of-gem5-build-time"></a><a class="link" href="#pybind11-accounts-for-50-of-gem5-build-time">35.2.3.3.1. pybind11 accounts for 50% of gem5 build time</a></h6>
<div class="paragraph">
<p><a href="https://gem5.atlassian.net/browse/GEM5-366" class="bare">https://gem5.atlassian.net/browse/GEM5-366</a></p>
</div>
<div class="paragraph">
<p>Yes, <a href="#pybind11">pybind11</a> is slow to build.</p>
</div>
<div class="paragraph">
<p>See also: <a href="#gem5-python-c-interaction">gem5 Python C++ interaction</a>.</p>
</div>
</div>
<div class="sect5">
<h6 id="benchmark-gem5-single-file-change-rebuild-time"><a class="anchor" href="#benchmark-gem5-single-file-change-rebuild-time"></a><a class="link" href="#benchmark-gem5-single-file-change-rebuild-time">35.2.3.3.2. Benchmark gem5 single file change rebuild time</a></h6>
<div class="paragraph">
<p>This is the critical development parameter, and is dominated by the link time of huge binaries.</p>
</div>
<div class="paragraph">
<p>In order to benchmark it better, make a comment only change to:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>vim submodules/gem5/src/sim/main.cc</pre>
</div>
</div>
<div class="paragraph">
<p>then rebuild with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --arch aarch64 --verbose</pre>
</div>
</div>
<div class="paragraph">
<p>and then copy the link command to a separate Bash file. Then you can time and modify it easily.</p>
</div>
<div class="paragraph">
<p>Some approximate reference values on <a href="#p51">2017 Lenovo ThinkPad P51</a> LKMC d4b3e064adeeace3c3e7d106801f95c14637c12f + 1 (doing multiple runs to warm up disk caches):</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>opt</code></p>
<div class="ulist">
<ul>
<li>
<p>unmodified: 10 seconds</p>
</li>
<li>
<p><code>LDFLAGS_EXTRA=-fuse-ld=gold</code>: 6 seconds. Huge improvement! Note that in general you have to do a full rebuild or else link may fail:  <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=23869" class="bare">https://sourceware.org/bugzilla/show_bug.cgi?id=23869</a></p>
<div class="paragraph">
<p>More info on gold:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/3476093/replacing-ld-with-gold-any-experience/53921263#53921263" class="bare">https://stackoverflow.com/questions/3476093/replacing-ld-with-gold-any-experience/53921263#53921263</a></p>
</li>
<li>
<p><a href="https://gem5-review.googlesource.com/c/public/gem5/+/14075" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/14075</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p><code>debug</code></p>
<div class="ulist">
<ul>
<li>
<p>unmodified: 14 seconds. Why so much slower than unmodified?</p>
</li>
<li>
<p><code>-fuse-ld=gold</code>: <code>internal error in read_cie, at ../../gold/ehframe.cc:919</code> on Ubuntu 18.04 all GCC. <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=23869" class="bare">https://sourceware.org/bugzilla/show_bug.cgi?id=23869</a></p>
</li>
</ul>
</div>
</li>
<li>
<p><code>fast</code></p>
<div class="ulist">
<ul>
<li>
<p><code>--force-lto</code>: 1 minute. Slower as expected, since more optimizations are done at link time. <code>--force-lto</code> is only used for <code>fast</code>, and it adds <code>-flto</code> to the build.</p>
</li>
</ul>
</div>
</li>
<li>
<p><code>opt LDFLAGS_EXTRA=-s</code>: stripping the executable greatly reduces link time, but you get no symbols</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>ramfs made no difference, the kernel must be caching files in memory very efficiently already.</p>
</div>
<div class="paragraph">
<p>In addition to the link time, scons startup time can also be considerable:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://gem5.atlassian.net/browse/GEM5-256" class="bare">https://gem5.atlassian.net/browse/GEM5-256</a></p>
</li>
<li>
<p><a href="https://gem5-review.googlesource.com/c/public/gem5/+/25385" class="bare">https://gem5-review.googlesource.com/c/public/gem5/+/25385</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>On LKMC 220c3a434499e4713664d4a47c246cb81ee0a06a gem5 63e96992568d8a8a0dccac477b8b7f1370ac7e98 (Sep 2020):</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>opt</code></p>
<div class="ulist">
<ul>
<li>
<p>default link: <code>18.32user 3.99system 0:22.33elapsed 99%CPU (0avgtext+0avgdata 4622908maxresident)k</code></p>
</li>
<li>
<p><code>LDFLAGS_EXTRA=-fuse-ld=lld</code> (after a build with default linker): <code>6.74user 1.81system 0:03.85elapsed 222%CPU (0avgtext+0avgdata 7025292maxresident)k</code></p>
</li>
<li>
<p><code>LDFLAGS_EXTRA=-fuse-ld=gold</code>: <code>7.70user 1.36system 0:09.44elapsed 95%CPU (0avgtext+0avgdata 5959152maxresident)k</code></p>
<div class="ulist">
<ul>
<li>
<p><code>LDFLAGS_EXTRA=-fuse-ld=gold -Wl,--threads -Wl,--thread-count=8</code>: <code>9.66user 1.86system 0:04.62elapsed 249%CPU (0avgtext+0avgdata 5989916maxresident)k</code></p>
<div class="paragraph">
<p>Arghhh, it does not use multile threads by default&#8230;&#8203; <a href="https://stackoverflow.com/questions/5142753/can-gcc-use-multiple-cores-when-linking/42302047#42302047" class="bare">https://stackoverflow.com/questions/5142753/can-gcc-use-multiple-cores-when-linking/42302047#42302047</a></p>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="benchmark-machines"><a class="anchor" href="#benchmark-machines"></a><a class="link" href="#benchmark-machines">35.3. Benchmark machines</a></h3>
<div class="sect3">
<h4 id="p51"><a class="anchor" href="#p51"></a><a class="link" href="#p51">35.3.1. 2017 Lenovo ThinkPad P51</a></h4>
<div class="paragraph">
<p>Serial number: TYPE 20HH-CTO1WW S/N PF-0V5V5N 17/11</p>
</div>
<div class="paragraph">
<p>Parts: <a href="https://support.lenovo.com/gb/en/solutions/pd105026" class="bare">https://support.lenovo.com/gb/en/solutions/pd105026</a> (<a href="https://web.archive.org/web/20200607133848/https://support.lenovo.com/gb/en/solutions/pd105026">archive</a>)</p>
</div>
<div class="paragraph">
<p>Hardware maintenance manual: <a href="https://download.lenovo.com/pccbbs/mobiles_pdf/p51_hmm_en_sp40k88791_01.pdf" class="bare">https://download.lenovo.com/pccbbs/mobiles_pdf/p51_hmm_en_sp40k88791_01.pdf</a> (<a href="https://web.archive.org/web/20200607155330/https://download.lenovo.com/pccbbs/mobiles_pdf/p51_hmm_en_sp40k88791_01.pdf">archive</a>)</p>
</div>
<div class="paragraph">
<p>Summary string of key hardware for copy paste:</p>
</div>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Lenovo ThinkPad P51 laptop with CPU: <a href="#intel-core-i7-7820hq-cpu">Intel Core i7-7820HQ</a> (4 cores / 8 threads, 2.90 GHz base, 8 MB cache), DRAM: 2x <a href="#samsung-m471a2k43bb1-crc-16gb-dram">Samsung M471A2K43BB1-CRC</a> (2x 16GiB, 2400 Mbps), SSD: <a href="#samsung-mzvlb512hajq-000l7-512gb-ssd">Samsung MZVLB512HAJQ-000L7</a> (512GB, 3,000 MB/s).</p>
</div>
</blockquote>
</div>
<div class="paragraph">
<p>Further specs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Hard disk: <a href="#seagate-st1000lm035-1rk1-1tb-hard-disk">Seagate ST1000LM035-1RK1 1TB hard disk</a></p>
</li>
<li>
<p>GPU: <a href="#nvidia-quadro-m1200-4gb-gddr5-gpu">NVIDIA Quadro M1200 4GB GDDR5 GPU</a></p>
</li>
<li>
<p>Pre-installed OS:</p>
<div class="ulist">
<ul>
<li>
<p>Windows 10 Pro 64</p>
</li>
<li>
<p>Windows 10 Pro 64 WE (EN/FR/DE/NL/IT)</p>
</li>
</ul>
</div>
</li>
<li>
<p>Display: 15.6" FHD (1920x1080), anti-glare, IPS</p>
</li>
<li>
<p>With Color Sensor</p>
</li>
<li>
<p>720p HD Camera with Microphone</p>
</li>
<li>
<p>Keyboard with Number Pad - Euro English</p>
</li>
<li>
<p>3+3BCP, Fingerprint Reader,Color Sensor</p>
</li>
<li>
<p>Integrated Fingerprint Reader</p>
</li>
<li>
<p>Hardware dTPM2.0 Enabled</p>
</li>
<li>
<p>1TB 5400rpm HDD</p>
</li>
<li>
<p>170W AC Adapter - UK(3pin)</p>
</li>
<li>
<p>6 Cell Li-Polymer Battery, 90Wh</p>
</li>
<li>
<p>Intel Dual Band Wireless AC(2x2) 8265, Bluetooth Version 4.1, vPro</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Parts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>keyboard FRU number: 01HW271 (written on part, Payton2Walter2 NBL KBD,USI,DFN according to <a href="https://support.lenovo.com/us/en/partslookup" class="bare">https://support.lenovo.com/us/en/partslookup</a> That website says 01ER981 is equivalent (Payton2Walter2 NBL KBD,USI,CHY), just different manufacturer</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Reddit threads:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.reddit.com/r/linux4noobs/comments/5zyejw/update_1604_tp_1610_boot_hangs_at_started_nvidia/" class="bare">https://www.reddit.com/r/linux4noobs/comments/5zyejw/update_1604_tp_1610_boot_hangs_at_started_nvidia/</a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/Lenovo/comments/6g8m9w/ubuntu_on_lenovo_p51/" class="bare">https://www.reddit.com/r/Lenovo/comments/6g8m9w/ubuntu_on_lenovo_p51/</a></p>
</li>
<li>
<p><a href="https://www.reddit.com/r/thinkpad/comments/6hi0zn/if_youre_thinking_of_running_linux_on_a_p51_read/" class="bare">https://www.reddit.com/r/thinkpad/comments/6hi0zn/if_youre_thinking_of_running_linux_on_a_p51_read/</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="p51-benchmarks"><a class="anchor" href="#p51-benchmarks"></a><a class="link" href="#p51-benchmarks">35.3.1.1. P51 benchmarks</a></h5>

</div>
<div class="sect4">
<h5 id="intel-core-i7-7820hq-cpu"><a class="anchor" href="#intel-core-i7-7820hq-cpu"></a><a class="link" href="#intel-core-i7-7820hq-cpu">35.3.1.2. Intel Core i7-7820HQ CPU</a></h5>
<div class="paragraph">
<p><a href="https://ark.intel.com/products/97496/Intel-Core-i7-7820HQ-Processor-8M-Cache-up-to-3-90-GHz-" class="bare">https://ark.intel.com/products/97496/Intel-Core-i7-7820HQ-Processor-8M-Cache-up-to-3-90-GHz-</a> (<a href="http://web.archive.org/web/20181224203737/https://ark.intel.com/products/97496/Intel-Core-i7-7820HQ-Processor-8M-Cache-up-to-3-90-GHz-">archive</a>).</p>
</div>
<div class="paragraph">
<p>Cache: 8MB</p>
</div>
<div class="paragraph">
<p>Max frequency: 3.90GHz</p>
</div>
<div class="paragraph">
<p>Cores: 4</p>
</div>
<div class="paragraph">
<p><a href="#hardware-threads">Hardware threads</a>: 8</p>
</div>
<div class="paragraph">
<p>Recommended customer price: 378.00 USD</p>
</div>
<div class="paragraph">
<p>Launch date: Q1'17</p>
</div>
<div class="paragraph">
<p>Process: 14 nm</p>
</div>
<div class="paragraph">
<p><code>cat /proc/cpuinfo</code> of one CPU on Ubuntu 20.04 Linux kernel 5.4.0:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 158
model name      : Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
stepping        : 9
microcode       : 0xd6
cpu MHz         : 1025.664
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds
bogomips        : 5799.77
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:</pre>
</div>
</div>
<div class="paragraph">
<p><code>getconf -a | grep CACHE</code> on Ubuntu 20.04 Linux kernel 5.4.0:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>LEVEL1_ICACHE_SIZE                 32768
LEVEL1_ICACHE_ASSOC                8
LEVEL1_ICACHE_LINESIZE             64
LEVEL1_DCACHE_SIZE                 32768
LEVEL1_DCACHE_ASSOC                8
LEVEL1_DCACHE_LINESIZE             64
LEVEL2_CACHE_SIZE                  262144
LEVEL2_CACHE_ASSOC                 4
LEVEL2_CACHE_LINESIZE              64
LEVEL3_CACHE_SIZE                  8388608
LEVEL3_CACHE_ASSOC                 16
LEVEL3_CACHE_LINESIZE              64
LEVEL4_CACHE_SIZE                  0
LEVEL4_CACHE_ASSOC                 0
LEVEL4_CACHE_LINESIZE              0</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="samsung-m471a2k43bb1-crc-16gb-dram"><a class="anchor" href="#samsung-m471a2k43bb1-crc-16gb-dram"></a><a class="link" href="#samsung-m471a2k43bb1-crc-16gb-dram">35.3.1.3. Samsung M471A2K43BB1-CRC 16GB DRAM</a></h5>
<div class="paragraph">
<p>Nominal speed: 2400 Mbps</p>
</div>
<div class="paragraph">
<p>Type: SODIMM</p>
</div>
<div class="paragraph">
<p><a href="https://www.samsung.com/semiconductor/dram/module/M471A2K43BB1-CRC/" class="bare">https://www.samsung.com/semiconductor/dram/module/M471A2K43BB1-CRC/</a> (<a href="http://web.archive.org/web/20181224202657/https://www.samsung.com/semiconductor/dram/module/M471A2K43BB1-CRC/">archive</a>).</p>
</div>
<div class="paragraph">
<p><a href="https://www.amazon.co.uk/Samsung-DDR4-16-GB-DDR4-2400-MHz-Memory-Module/dp/B016N24XKQ" class="bare">https://www.amazon.co.uk/Samsung-DDR4-16-GB-DDR4-2400-MHz-Memory-Module/dp/B016N24XKQ</a> (<a href="http://web.archive.org/web/20181224203214/https://www.amazon.co.uk/Samsung-DDR4-16-GB-DDR4-2400-MHz-Memory-Module/dp/B016N24XKQ">archive</a>) 355.43 UK Pounds for 2x 16 GiB.</p>
</div>
</div>
<div class="sect4">
<h5 id="samsung-mzvlb512hajq-000l7-512gb-ssd"><a class="anchor" href="#samsung-mzvlb512hajq-000l7-512gb-ssd"></a><a class="link" href="#samsung-mzvlb512hajq-000l7-512gb-ssd">35.3.1.4. Samsung MZVLB512HAJQ-000L7 512GB SSD</a></h5>
<div class="paragraph">
<p>PCIe TLC OPAL2.</p>
</div>
<div class="paragraph">
<p><a href="https://www.samsung.com/semiconductor/ssd/client-ssd/MZVLB512HAJQ/" class="bare">https://www.samsung.com/semiconductor/ssd/client-ssd/MZVLB512HAJQ/</a> (<a href="http://web.archive.org/web/20181224225400/https://www.samsung.com/semiconductor/ssd/client-ssd/MZVLB512HAJQ/">archive</a>).</p>
</div>
<div class="paragraph">
<p><a href="https://www.samsung.com/semiconductor/global.semi/file/resource/2018/05/PM981_M.2_SSD_Datasheet_v1.3_for_General.pdf" class="bare">https://www.samsung.com/semiconductor/global.semi/file/resource/2018/05/PM981_M.2_SSD_Datasheet_v1.3_for_General.pdf</a> | <a href="http://web.archive.org/web/20181224225410/https://www.samsung.com/semiconductor/global.semi/file/resource/2018/05/PM981_M.2_SSD_Datasheet_v1.3_for_General.pdf" class="bare">http://web.archive.org/web/20181224225410/https://www.samsung.com/semiconductor/global.semi/file/resource/2018/05/PM981_M.2_SSD_Datasheet_v1.3_for_General.pdf</a></p>
</div>
<div class="paragraph">
<p><code>sudo hdparm -Tt /dev/nvme0n1p5</code> on Ubuntu 20.04:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> Timing cached reads:   29812 MB in  1.99 seconds = 15007.00 MB/sec
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 6328 MB in  3.00 seconds = 2109.00 MB/sec</pre>
</div>
</div>
<div class="paragraph">
<p>Nominal maximum sequential read speed: 3,000 MB/s</p>
</div>
</div>
<div class="sect4">
<h5 id="seagate-st1000lm035-1rk1-1tb-hard-disk"><a class="anchor" href="#seagate-st1000lm035-1rk1-1tb-hard-disk"></a><a class="link" href="#seagate-st1000lm035-1rk1-1tb-hard-disk">35.3.1.5. Seagate ST1000LM035-1RK1 1TB hard disk</a></h5>
<div class="paragraph">
<p>1TB.</p>
</div>
<div class="paragraph">
<p><a href="https://www.disctech.com/Seagate-ST1000LM035-1TB-SATA-Hard-Drive" class="bare">https://www.disctech.com/Seagate-ST1000LM035-1TB-SATA-Hard-Drive</a> 80 USD | <a href="http://web.archive.org/web/20181224201408/https://www.disctech.com/Seagate-ST1000LM035-1TB-SATA-Hard-Drive" class="bare">http://web.archive.org/web/20181224201408/https://www.disctech.com/Seagate-ST1000LM035-1TB-SATA-Hard-Drive</a></p>
</div>
<div class="paragraph">
<p><a href="https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf" class="bare">https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf</a> | <a href="http://web.archive.org/web/20181225095438/https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf" class="bare">http://web.archive.org/web/20181225095438/https://www.seagate.com/www-content/datasheets/pdfs/mobile-hddDS1861-2-1603-en_US.pdf</a></p>
</div>
<div class="paragraph">
<p><code>sudo hdparm -Tt /dev/sda3</code> on Ubuntu 20.04:</p>
</div>
<div class="literalblock">
<div class="content">
<pre> Timing cached reads:   29594 MB in  1.99 seconds = 14893.89 MB/sec
 Timing buffered disk reads: 386 MB in  3.01 seconds = 128.07 MB/sec</pre>
</div>
</div>
<div class="paragraph">
<p>Nominal maximum speed: 140MB/s</p>
</div>
</div>
<div class="sect4">
<h5 id="nvidia-quadro-m1200-4gb-gddr5-gpu"><a class="anchor" href="#nvidia-quadro-m1200-4gb-gddr5-gpu"></a><a class="link" href="#nvidia-quadro-m1200-4gb-gddr5-gpu">35.3.1.6. NVIDIA Quadro M1200 4GB GDDR5 GPU</a></h5>

</div>
</div>
</div>
<div class="sect2">
<h3 id="benchmark-internets"><a class="anchor" href="#benchmark-internets"></a><a class="link" href="#benchmark-internets">35.4. Benchmark Internets</a></h3>
<div class="sect3">
<h4 id="38mbps-internet"><a class="anchor" href="#38mbps-internet"></a><a class="link" href="#38mbps-internet">35.4.1. 38Mbps internet</a></h4>
<div class="paragraph">
<p>2c12b21b304178a81c9912817b782ead0286d282:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>shallow clone of all submodules: 4 minutes.</p>
</li>
<li>
<p><code>make source</code>: 2 minutes</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Google M-lab speed test: 36.4Mbps</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="benchmark-this-repo-bibliography"><a class="anchor" href="#benchmark-this-repo-bibliography"></a><a class="link" href="#benchmark-this-repo-bibliography">35.5. Benchmark this repo bibliography</a></h3>
<div class="paragraph">
<p>gem5:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://www.mail-archive.com/gem5-users@gem5.org/msg15262.html" class="bare">https://www.mail-archive.com/gem5-users@gem5.org/msg15262.html</a> which parts of the gem5 code make it slow</p>
</li>
<li>
<p>what are the minimum system requirements:</p>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/47997565/gem5-system-requirements-for-decent-performance/48941793#48941793" class="bare">https://stackoverflow.com/questions/47997565/gem5-system-requirements-for-decent-performance/48941793#48941793</a></p>
</li>
<li>
<p><a href="https://github.com/gem5/gem5/issues/25" class="bare">https://github.com/gem5/gem5/issues/25</a></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="compilers"><a class="anchor" href="#compilers"></a><a class="link" href="#compilers">36. Compilers</a></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Argh, compilers are boring, let&#8217;s learn a bit about them.</p>
</div>
<div class="sect2">
<h3 id="prevent-statement-reordering"><a class="anchor" href="#prevent-statement-reordering"></a><a class="link" href="#prevent-statement-reordering">36.1. Prevent statement reordering</a></h3>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/prevent_reorder.cpp">userland/gcc/prevent_reorder.cpp</a></p>
</div>
<div class="paragraph">
<p><a href="https://stackoverflow.com/questions/37786547/enforcing-statement-order-in-c/56865717#56865717" class="bare">https://stackoverflow.com/questions/37786547/enforcing-statement-order-in-c/56865717#56865717</a></p>
</div>
<div class="paragraph">
<p>We often need to do this to be sure that benchmark instrumentation is actually being put around the region of interest!</p>
</div>
</div>
<div class="sect2">
<h3 id="c-busy-loop"><a class="anchor" href="#c-busy-loop"></a><a class="link" href="#c-busy-loop">36.2. C busy loop</a></h3>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gcc/busy_loop.c">userland/gcc/busy_loop.c</a></p>
</div>
<div class="paragraph">
<p>The hard part is how to prevent the compiler from optimizing it away: <a href="https://stackoverflow.com/questions/7083482/how-to-prevent-gcc-from-optimizing-out-a-busy-wait-loop/58758133#58758133" class="bare">https://stackoverflow.com/questions/7083482/how-to-prevent-gcc-from-optimizing-out-a-busy-wait-loop/58758133#58758133</a></p>
</div>
<div class="paragraph">
<p><a href="#disas">Disassembly</a> analysis:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --userland userland/gcc/busy_loop.out busy_loop</pre>
</div>
</div>
<div class="paragraph">
<p>which contains at LKMC eb22fd3b6e7fff7e9ef946a88b208debf5b419d5:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>10      ) {
   0x0000000000400700 &lt;+0&gt;:     ff 83 00 d1     sub     sp, sp, #0x20
   0x0000000000400704 &lt;+4&gt;:     e0 07 00 f9     str     x0, [sp, #8]
   0x0000000000400708 &lt;+8&gt;:     e1 03 00 f9     str     x1, [sp]

11          for (unsigned long long i = 0; i &lt; max2; i++) {
   0x000000000040070c &lt;+12&gt;:    ff 0f 00 f9     str     xzr, [sp, #24]
   0x0000000000400710 &lt;+16&gt;:    11 00 00 14     b       0x400754 &lt;busy_loop+84&gt;

12              for (unsigned long long j = 0; j &lt; max; j++) {
   0x0000000000400714 &lt;+20&gt;:    ff 0b 00 f9     str     xzr, [sp, #16]
   0x0000000000400718 &lt;+24&gt;:    08 00 00 14     b       0x400738 &lt;busy_loop+56&gt;

13                  __asm__ __volatile__ ("" : "+g" (i), "+g" (j) : :);
   0x000000000040071c &lt;+28&gt;:    e1 0f 40 f9     ldr     x1, [sp, #24]
   0x0000000000400720 &lt;+32&gt;:    e0 0b 40 f9     ldr     x0, [sp, #16]
   0x0000000000400724 &lt;+36&gt;:    e1 0f 00 f9     str     x1, [sp, #24]
   0x0000000000400728 &lt;+40&gt;:    e0 0b 00 f9     str     x0, [sp, #16]

12              for (unsigned long long j = 0; j &lt; max; j++) {
   0x000000000040072c &lt;+44&gt;:    e0 0b 40 f9     ldr     x0, [sp, #16]
   0x0000000000400730 &lt;+48&gt;:    00 04 00 91     add     x0, x0, #0x1
   0x0000000000400734 &lt;+52&gt;:    e0 0b 00 f9     str     x0, [sp, #16]
   0x0000000000400738 &lt;+56&gt;:    e1 0b 40 f9     ldr     x1, [sp, #16]
   0x000000000040073c &lt;+60&gt;:    e0 07 40 f9     ldr     x0, [sp, #8]
   0x0000000000400740 &lt;+64&gt;:    3f 00 00 eb     cmp     x1, x0
   0x0000000000400744 &lt;+68&gt;:    c3 fe ff 54     b.cc    0x40071c &lt;busy_loop+28&gt;  // b.lo, b.ul, b.last

11          for (unsigned long long i = 0; i &lt; max2; i++) {
   0x0000000000400748 &lt;+72&gt;:    e0 0f 40 f9     ldr     x0, [sp, #24]
   0x000000000040074c &lt;+76&gt;:    00 04 00 91     add     x0, x0, #0x1
   0x0000000000400750 &lt;+80&gt;:    e0 0f 00 f9     str     x0, [sp, #24]
   0x0000000000400754 &lt;+84&gt;:    e1 0f 40 f9     ldr     x1, [sp, #24]
   0x0000000000400758 &lt;+88&gt;:    e0 03 40 f9     ldr     x0, [sp]
   0x000000000040075c &lt;+92&gt;:    3f 00 00 eb     cmp     x1, x0
   0x0000000000400760 &lt;+96&gt;:    a3 fd ff 54     b.cc    0x400714 &lt;busy_loop+20&gt;  // b.lo, b.ul, b.last

14              }
15          }
16      }
   0x0000000000400764 &lt;+100&gt;:   1f 20 03 d5     nop
   0x0000000000400768 &lt;+104&gt;:   ff 83 00 91     add     sp, sp, #0x20
   0x000000000040076c &lt;+108&gt;:   c0 03 5f d6     ret</pre>
</div>
</div>
<div class="paragraph">
<p>We look for the internal backwards jumps, and we find two:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>   0x00000000004006dc &lt;+68&gt;:    c8 fe ff 54     b.hi    0x4006b4 &lt;busy_loop+28&gt;  // b.pmore
   0x00000000004006f8 &lt;+96&gt;:    a8 fd ff 54     b.hi    0x4006ac &lt;busy_loop+20&gt;  // b.pmore</pre>
</div>
</div>
<div class="paragraph">
<p>and so clearly the one at 0x4006dc happens first and jumps to a larger address than the other one, so the internal loop must be between 4006dc and 4006b4, which contains exactly 11 instructions.</p>
</div>
<div class="paragraph">
<p>Oh my God, unoptimized code is so horrendously inefficient, even I can&#8217;t stand all those useless loads and stores to memory variables!!!</p>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="computer-architecture"><a class="anchor" href="#computer-architecture"></a><a class="link" href="#computer-architecture">37. Computer architecture</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="instruction-pipelining"><a class="anchor" href="#instruction-pipelining"></a><a class="link" href="#instruction-pipelining">37.1. Instruction pipelining</a></h3>
<div class="paragraph">
<p>In gem5, can be seen on:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#gem5-minorcpu">gem5 MinorCPU</a></p>
</li>
<li>
<p><a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a></p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="classic-risc-pipeline"><a class="anchor" href="#classic-risc-pipeline"></a><a class="link" href="#classic-risc-pipeline">37.1.1. Classic RISC pipeline</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Classic_RISC_pipeline" class="bare">https://en.wikipedia.org/wiki/Classic_RISC_pipeline</a></p>
</div>
<div class="paragraph">
<p>gem5&#8217;s <a href="#gem5-minorcpu">gem5 MinorCPU</a> implements a similar but 4 stage pipeline. TODO why didn&#8217;t they go with the classic RISC pipeline instead?</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="superscalar-processor"><a class="anchor" href="#superscalar-processor"></a><a class="link" href="#superscalar-processor">37.2. Superscalar processor</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Superscalar_processor" class="bare">https://en.wikipedia.org/wiki/Superscalar_processor</a></p>
</div>
<div class="paragraph">
<p><a href="http://www.lighterra.com/papers/modernmicroprocessors/" class="bare">http://www.lighterra.com/papers/modernmicroprocessors/</a> explains it well.</p>
</div>
<div class="paragraph">
<p>You basically decode multiple instructions in one go, and run them at the same time if they can go in separate <a href="#execution-unit">functional units</a> and have no conflicts. Genius!</p>
</div>
<div class="paragraph">
<p>And so the concept of <a href="#branch-predictor">branch predictor</a> must come in here: when a conditional branch is reached, you have to decide which side to execute before knowing for sure.</p>
</div>
<div class="paragraph">
<p>This is why it is called a type of <a href="#instruction-level-parallelism">Instruction level parallelism</a>.</p>
</div>
<div class="paragraph">
<p>Although this is a microarchitectural feature, it is so important that it is publicly documented. For example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/ARM_Cortex-A77" class="bare">https://en.wikipedia.org/wiki/ARM_Cortex-A77</a>: ARM Cortex A77 (2019) has a 4-wide superscalar decode (and is <a href="#out-of-order-execution">out-of-order</a>)</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="execution-unit"><a class="anchor" href="#execution-unit"></a><a class="link" href="#execution-unit">37.2.1. Execution unit</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Execution_unit" class="bare">https://en.wikipedia.org/wiki/Execution_unit</a></p>
</div>
<div class="paragraph">
<p>gem5 calls them "functional units".</p>
</div>
<div class="paragraph">
<p>gem5 has <a href="#execution-unit">functional units</a> explicitly modelled as shown at <a href="#gem5-functional-units">gem5 functional units</a>, and those are used by both <a href="#gem5-minorcpu">gem5 MinorCPU</a> and <a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a>.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="out-of-order-execution"><a class="anchor" href="#out-of-order-execution"></a><a class="link" href="#out-of-order-execution">37.3. Out-of-order execution</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Out-of-order_execution" class="bare">https://en.wikipedia.org/wiki/Out-of-order_execution</a></p>
</div>
<div class="paragraph">
<p>gem5&#8217;s model is <a href="#gem5-derivo3cpu">gem5 <code>DerivO3CPU</code></a>.</p>
</div>
<div class="paragraph">
<p>Allows working around data dependencies: you can execute the second next instruction forward if the first next depends on the current one.</p>
</div>
<div class="paragraph">
<p>Likely used on basically all (?) 2020 non-power-constrained CPUs.</p>
</div>
<div class="paragraph">
<p>As mentioned at: <a href="https://stackoverflow.com/questions/10074831/what-is-general-difference-between-superscalar-and-ooo-execution" class="bare">https://stackoverflow.com/questions/10074831/what-is-general-difference-between-superscalar-and-ooo-execution</a> it is in theory possible for an out-of-order CPU to not a <a href="#superscalar-processor">Superscalar processor</a>, but the combination is so natural (since you can look ahead, you might as well run it!) that it is not super common.</p>
</div>
<div class="sect3">
<h4 id="speculative-execution"><a class="anchor" href="#speculative-execution"></a><a class="link" href="#speculative-execution">37.3.1. Speculative execution</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Speculative_execution" class="bare">https://en.wikipedia.org/wiki/Speculative_execution</a></p>
</div>
<div class="paragraph">
<p>A gem5 example can be seen at: <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative</a>.</p>
</div>
<div class="paragraph">
<p>Bibliography:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/49601910/out-of-order-execution-vs-speculative-execution" class="bare">https://stackoverflow.com/questions/49601910/out-of-order-execution-vs-speculative-execution</a></p>
</li>
</ul>
</div>
<div class="sect4">
<h5 id="branch-predictor"><a class="anchor" href="#branch-predictor"></a><a class="link" href="#branch-predictor">37.3.1.1. Branch predictor</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Branch_predictor" class="bare">https://en.wikipedia.org/wiki/Branch_predictor</a></p>
</div>
<div class="paragraph">
<p>Comes in for <a href="#superscalar-processor">superscalar processors</a>.</p>
</div>
<div class="paragraph">
<p>A gem5 example can be seen at: <a href="#gem5-event-queue-derivo3cpu-syscall-emulation-freestanding-example-analysis-speculative">gem5 event queue DerivO3CPU syscall emulation freestanding example analysis: speculative</a>.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="re-order-buffer"><a class="anchor" href="#re-order-buffer"></a><a class="link" href="#re-order-buffer">37.3.2. Re-order buffer</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Re-order_buffer" class="bare">https://en.wikipedia.org/wiki/Re-order_buffer</a></p>
</div>
</div>
<div class="sect3">
<h4 id="register-renaming"><a class="anchor" href="#register-renaming"></a><a class="link" href="#register-renaming">37.3.3. Register renaming</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Register_renaming" class="bare">https://en.wikipedia.org/wiki/Register_renaming</a></p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="instruction-level-parallelism"><a class="anchor" href="#instruction-level-parallelism"></a><a class="link" href="#instruction-level-parallelism">37.4. Instruction level parallelism</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Instruction-level_parallelism" class="bare">https://en.wikipedia.org/wiki/Instruction-level_parallelism</a></p>
</div>
<div class="paragraph">
<p>Basically means decoding and then potentially executing a bunch of instructions in one go.</p>
</div>
<div class="paragraph">
<p>Important examples:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#superscalar-processor">Superscalar processor</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="hardware-threads"><a class="anchor" href="#hardware-threads"></a><a class="link" href="#hardware-threads">37.5. Hardware threads</a></h3>
<div class="paragraph">
<p>Intel name: "Hyperthreading"</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://superuser.com/questions/133082/what-is-the-difference-between-hyper-threading-and-multiple-cores/995858#995858" class="bare">https://superuser.com/questions/133082/what-is-the-difference-between-hyper-threading-and-multiple-cores/995858#995858</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/5593328/software-threads-vs-hardware-threads/61415402#61415402" class="bare">https://stackoverflow.com/questions/5593328/software-threads-vs-hardware-threads/61415402#61415402</a></p>
</li>
<li>
<p><a href="https://superuser.com/questions/122536/what-is-hyper-threading-and-how-does-it-work" class="bare">https://superuser.com/questions/122536/what-is-hyper-threading-and-how-does-it-work</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>gem5 appears to possibly have attempted to implement hardware threads in <a href="#gem5-syscall-emulation-mode">gem5 syscall emulation mode</a> as mentioned at <a href="#gem5-syscall-emulation-smt">gem5 syscall emulation --smt</a>.</p>
</div>
<div class="paragraph">
<p>On fs.py it is not exposed in any in-tree config however, and as pointed by the above issue O3 FS has an assert that prevents it in <a href="https://github.com/gem5/gem5/blob/377898c4034c72b84b2662ed252fa25079a4ea62/src/cpu/o3/cpu.cc#L313">src/cpu/o3/cpu.cc</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>            // SMT is not supported in FS mode yet.
            assert(this-&gt;numThreads == 1);</pre>
</div>
</div>
<div class="paragraph">
<p>TODO why only in fs.py? Is there much difference between fs and se from a hyperthreading point of view? Maybe the message is there because as concluded in <a href="#gem5-o3threadcontext">gem5 <code>O3ThreadContext</code></a>, registeres for <code>DerivO3CPU</code> are stored in <code>DerivO3CPU</code> itself (<code>FullO3CPU</code>), and therefore there is no way to to currently represent multiple register sets per CPU.</p>
</div>
<div class="paragraph">
<p>Other CPUs just appear to fail non-gracefully, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 -- --param 'system.cpu[0].numThreads = 2'</pre>
</div>
</div>
<div class="paragraph">
<p>fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>fatal: fatal condition interrupts.size() != numThreads occurred: CPU system.cpu has 1 interrupt controllers, but is expecting one per thread (2)</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="caches"><a class="anchor" href="#caches"></a><a class="link" href="#caches">37.6. Caches</a></h3>
<div class="paragraph">
<p><a href="https://courses.cs.washington.edu/courses/cse378/09wi/lectures/lec15.pdf" class="bare">https://courses.cs.washington.edu/courses/cse378/09wi/lectures/lec15.pdf</a> contains some of the first pictures you should see.</p>
</div>
<div class="paragraph">
<p>In a direct-mapped cache architecture (every address has a single possible block), a memory address can be broken up into:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>+-----+-------+--------------+
|     |       |              | full address
+-----+-------+--------------+
|     |       |              |
| tag | index | block offset |</pre>
</div>
</div>
<div class="paragraph">
<p>where:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>index: determines in which block the address will go. This is the "index/ID of the block" it will go into!</p>
</li>
<li>
<p>tag: allows us to differentiate between multiple addresses that have the same index</p>
<div class="paragraph">
<p>We really want tag to be the higher bits, so that consecutive blocks may be placed in the cache at once.</p>
</div>
</li>
<li>
<p>block offset: address withing the cache. Not used to find caches at all! Only used to find the data within the cache line</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If the cache is set associative, we just simply make the index smaller and add a bits to the tag.</p>
</div>
<div class="paragraph">
<p>For example, for a 2-way associative cache, we remove on bit from the index, and add it to the tag.</p>
</div>
<div class="sect3">
<h4 id="cache-coherence"><a class="anchor" href="#cache-coherence"></a><a class="link" href="#cache-coherence">37.6.1. Cache coherence</a></h4>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Cache_coherence" class="bare">https://en.wikipedia.org/wiki/Cache_coherence</a></p>
</div>
<div class="paragraph">
<p>In simple terms, when a certain group of caches of different CPUs are coherent, reads on one core always see the writes previously made by other cores. TODO: is it that strict, or just ordering? TODO what about simultaneous read and writes?</p>
</div>
<div class="paragraph">
<p><a href="http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf" class="bare">http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf</a> mentions that:</p>
</div>
<div class="paragraph">
<p>Cache coherence:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>guarantees eventual write propagation</p>
</li>
<li>
<p>guarantees a single order of all writes to same location</p>
</li>
<li>
<p>no guarantees on when writes propagate</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>And notably it contrasts that with <a href="#memory-consistency">Memory consistency</a>, which according to them is about ordering requirements on <em>different</em> addresses.</p>
</div>
<div class="paragraph">
<p>Algorithms to keep the caches of different cores of a system coherent. Only matters for multicore systems.</p>
</div>
<div class="paragraph">
<p>The main goal of such systems is to reduce the number of messages that have to be sent on the coherency bus, and even more importantly, to memory (which passes first through the coherency bus).</p>
</div>
<div class="paragraph">
<p>The main software use case example to have in mind is that of multiple threads incrementing an atomic counter as in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/cpp/atomic/std_atomic.cpp">userland/cpp/atomic/std_atomic.cpp</a>, see also: <a href="#atomic-cpp">atomic.cpp</a>. Then, if one processors writes to the cache, other processors have to know about it before they read from that address.</p>
</div>
<div class="paragraph">
<p>Even if caches are coherent, this is still not enough to avoid data race conditions, because this does not enforce atomicity of read modify write sequences. This is for example shown at: <a href="#detailed-gem5-analysis-of-how-data-races-happen">Detailed gem5 analysis of how data races happen</a>.</p>
</div>
<div class="sect4">
<h5 id="memory-consistency"><a class="anchor" href="#memory-consistency"></a><a class="link" href="#memory-consistency">37.6.1.1. Memory consistency</a></h5>
<div class="paragraph">
<p>According to <a href="http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf" class="bare">http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf</a> "memory consistency" is about ordering requirements of different memory addresses.</p>
</div>
<div class="paragraph">
<p>This is represented explicitly in C++ for example <a href="#cpp-memory-order">C++ std::memory_order</a>.</p>
</div>
<div class="sect5">
<h6 id="sequential-consistency"><a class="anchor" href="#sequential-consistency"></a><a class="link" href="#sequential-consistency">37.6.1.1.1. Sequential Consistency</a></h6>
<div class="paragraph">
<p>According to <a href="http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf" class="bare">http://www.inf.ed.ac.uk/teaching/courses/pa/Notes/lecture07-sc.pdf</a>, the strongest possible consistency, everything nicely ordered as you&#8217;d expect.</p>
</div>
</div>
</div>
<div class="sect4">
<h5 id="can-caches-snoop-data-from-other-caches"><a class="anchor" href="#can-caches-snoop-data-from-other-caches"></a><a class="link" href="#can-caches-snoop-data-from-other-caches">37.6.1.2. Can caches snoop data from other caches?</a></h5>
<div class="paragraph">
<p>Either they can snoop only control, or both control and data can be snooped.</p>
</div>
<div class="paragraph">
<p>The answer to this determines if some of the following design decisions make sense.</p>
</div>
<div class="paragraph">
<p>This is the central point in question at: <a href="https://electronics.stackexchange.com/questions/484830/why-is-a-flush-needed-in-the-msi-cache-coherency-protocol-when-moving-from-modif" class="bare">https://electronics.stackexchange.com/questions/484830/why-is-a-flush-needed-in-the-msi-cache-coherency-protocol-when-moving-from-modif</a></p>
</div>
<div class="paragraph">
<p>If data snoops are not possible, then data must always to to DRAM first.</p>
</div>
</div>
<div class="sect4">
<h5 id="vi-cache-coherence-protocol"><a class="anchor" href="#vi-cache-coherence-protocol"></a><a class="link" href="#vi-cache-coherence-protocol">37.6.1.3. VI cache coherence protocol</a></h5>
<div class="paragraph">
<p>Mentioned at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://courses.csail.mit.edu/6.888/spring13/lectures/L7-coherence.pdf" class="bare">http://courses.csail.mit.edu/6.888/spring13/lectures/L7-coherence.pdf</a></p>
</li>
<li>
<p><a href="http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf" class="bare">http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This is the most trivial, but likely it is too bad and most sources don&#8217;t even mention it.</p>
</div>
<div class="paragraph">
<p>In what follows I make some stuff up with design choice comparisons, needs confirmation.</p>
</div>
<div class="paragraph">
<p>In this protocol, every cache only needs a single bit of state: validity.</p>
</div>
<div class="paragraph">
<p>At the start, everything is invalid.</p>
</div>
<div class="paragraph">
<p>Then, when you need to read and are invalid, you send a read on bus. If there is another valid cache in another CPU, it services the request. Otherwise, goes the request goes to memory. After read you become valid.</p>
</div>
<div class="paragraph">
<p>Read for valid generates no bus requests, which is good.</p>
</div>
<div class="paragraph">
<p>When you write, if you are invalid, you must first read to get the full cache line, like for any other protocol.</p>
</div>
<div class="paragraph">
<p>Then, there are two possible design choices, either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>that read is marked as exclusive, and all caches that had it, snoop it become invalid.</p>
<div class="paragraph">
<p>Upside: no need to send the new data to the bus.</p>
</div>
<div class="paragraph">
<p>Downside: more invalidations. But those are not too serious, because future invalid reads tend to just hit the remaining valid cache.</p>
</div>
</li>
<li>
<p>after the read and write, you send the data on the bus, and those that had it update and become valid.</p>
<div class="paragraph">
<p>Downside: much more data on bus, so likely this is not going to be the best choice.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>So we take the first option.</p>
</div>
<div class="paragraph">
<p>When you write and are valid, you don&#8217;t need to read. But you still have invalidate everyone else, because multiple reads can lead to multiple valid holders, otherwise other valid holders would keep reading old values.</p>
</div>
<div class="paragraph">
<p>We could either do this with an exclusive read, and ignore the return, or with a new Invalidate request that has no reply. This invalidation is called <code>BusUpgr</code> to match with Wikipedia.</p>
</div>
<div class="paragraph">
<p>Write also has two other possible design choices, either:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>every write writes through to memory. This is likely never the best option.</p>
</li>
<li>
<p>when the cache is full, eviction leads to a write to memory.</p>
<div class="paragraph">
<p>If multiple valid holders may exist, then this may lead to multiple write through evictions of the same thing.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>So we take the second option.</p>
</div>
<div class="paragraph">
<p>With this we would have:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>V</p>
<div class="ulist">
<ul>
<li>
<p>PrRd</p>
<div class="ulist">
<ul>
<li>
<p>V
<strong>*</strong></p>
</li>
</ul>
</div>
</li>
<li>
<p>PrWr</p>
<div class="ulist">
<ul>
<li>
<p>V</p>
</li>
<li>
<p>BusUpgr</p>
</li>
</ul>
</div>
</li>
<li>
<p>BusRd</p>
<div class="ulist">
<ul>
<li>
<p>V</p>
</li>
<li>
<p>BusData</p>
</li>
</ul>
</div>
</li>
<li>
<p>BusRdX</p>
<div class="ulist">
<ul>
<li>
<p>I</p>
</li>
<li>
<p>BusData</p>
</li>
</ul>
</div>
</li>
<li>
<p>BusUpgr</p>
<div class="ulist">
<ul>
<li>
<p>I
<strong>*</strong></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>I</p>
<div class="ulist">
<ul>
<li>
<p>PrRd</p>
<div class="ulist">
<ul>
<li>
<p>V</p>
</li>
<li>
<p>BusRd</p>
</li>
</ul>
</div>
</li>
<li>
<p>PrWr</p>
<div class="ulist">
<ul>
<li>
<p>V</p>
</li>
<li>
<p>BusRdX</p>
</li>
</ul>
</div>
</li>
<li>
<p>BusRd</p>
<div class="ulist">
<ul>
<li>
<p>I
<strong>*</strong></p>
</li>
</ul>
</div>
</li>
<li>
<p>BusRdX</p>
<div class="ulist">
<ul>
<li>
<p>I
<strong>*</strong></p>
</li>
</ul>
</div>
</li>
<li>
<p>BusUpgr</p>
<div class="ulist">
<ul>
<li>
<p>I
<strong>*</strong></p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>Here Flush and BusData replies are omitted since those never lead to a change of state, nor to the sending of further messages.</p>
</div>
<div class="paragraph">
<p>TODO at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://courses.csail.mit.edu/6.888/spring13/lectures/L7-coherence.pdf" class="bare">http://courses.csail.mit.edu/6.888/spring13/lectures/L7-coherence.pdf</a></p>
</li>
<li>
<p><a href="http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf" class="bare">http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>why PrWr stays in invalid? Why do writes always go to memory? Why not wait until eviction?</p>
</div>
</div>
<div class="sect4">
<h5 id="msi-cache-coherence-protocol"><a class="anchor" href="#msi-cache-coherence-protocol"></a><a class="link" href="#msi-cache-coherence-protocol">37.6.1.4. MSI cache coherence protocol</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/MSI_protocol" class="bare">https://en.wikipedia.org/wiki/MSI_protocol</a></p>
</div>
<div class="paragraph">
<p>This is the most basic non-trivial coherency protocol, and therefore the first one you should learn.</p>
</div>
<div class="paragraph">
<p>Compared to the <a href="#vi-cache-coherence-protocol">VI cache coherence protocol</a>, MSI:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>adds one bit of knowledge per cache line (shared)</p>
</li>
<li>
<p>splits Valid into Modified and Shared depending on the shared bit</p>
</li>
<li>
<p>this allows us to not send BusUpgr messages on the bus when writing to Modified, since we now we know that the data is not present in any other cache!</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Helpful video: <a href="https://www.youtube.com/watch?v=gAUVAel-2Fg" class="bare">https://www.youtube.com/watch?v=gAUVAel-2Fg</a> "MSI Coherence - Georgia Tech - HPCA: Part 5" by Udacity.</p>
</div>
<div class="paragraph">
<p>Let&#8217;s focus on a single cache line representing a given memory address.</p>
</div>
<div class="paragraph">
<p>The system looks like this:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>+----+
|DRAM|
+----+
^
|
v
+--------+
| BUS    |
+--------+
^        ^
|        |
v        v
+------+ +------+
|CACHE1| |CACHE2|
+------+ +------+
^        ^
|        |
|        |
+----+   +----+
|CPU1|   |CPU2|
+----+   +----+</pre>
</div>
</div>
<div class="paragraph">
<p>MSI stands for which states each cache can be in for a given cache line. The states are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Modified: a single cache has the valid data and it has been modified from DRAM.</p>
<div class="paragraph">
<p>Both reads and writes are free, because we don&#8217;t have to worry about other processors.</p>
</div>
</li>
<li>
<p>Shared: the data is synchronized with DRAM, and may be present in multiple caches.</p>
<div class="paragraph">
<p>Reads are free, but writes need to do extra work.</p>
</div>
<div class="paragraph">
<p>This is the "most interesting" state of the protocol, as it allows for those free reads, even when multiple processors are using some address.</p>
</div>
</li>
<li>
<p>Invalid: the cache does not have the data, CPU reads and writes need to do extra work</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The above allowed states can be summarized in the following table:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>         CACHE1
         MSI
       M nny
CACHE2 S nyy
       I yyy</pre>
</div>
</div>
<div class="paragraph">
<p>The whole goal of the protocol is to maintain that state at all times, so that we can get those free reads when in shared state!</p>
</div>
<div class="paragraph">
<p>To do so, the caches have to pass messages between themselves! This means generating bus traffic, which has a cost and must be kept to a minimum.</p>
</div>
<div class="paragraph">
<p>The system components can receive and send the following messages:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>CPUn can send to CACHEn:</p>
<div class="ulist">
<ul>
<li>
<p>"Local read": CPU reads from cache</p>
</li>
<li>
<p>"Local write": CPU writes to cache</p>
</li>
</ul>
</div>
</li>
<li>
<p>CACHEn to itself:</p>
<div class="ulist">
<ul>
<li>
<p>"Evict": the cache is running out of space due to another request</p>
</li>
</ul>
</div>
</li>
<li>
<p>CACHEn can send the following message to the bus.</p>
<div class="ulist">
<ul>
<li>
<p>"Bus read": the cache needs to get the data. The reply will contain the full data line. It can come either from another cache that has the data, or from DRAM if none do.</p>
</li>
<li>
<p>"Bus write": the cache wants to modify some data, and it does not have the line.</p>
<div class="paragraph">
<p>The reply must contain the full data line, because maybe the processor just wants to change one byte, but the line is much larger.</p>
</div>
<div class="paragraph">
<p>That&#8217;s why this request can also be called "Read Exclusive", as it is basically a "Bus Read" + "Invalidate" in one</p>
</div>
</li>
<li>
<p>"Invalidate": the cache wants to modify some data, but it knows that all other caches are up to date, because it is in shared state.</p>
<div class="paragraph">
<p>Therefore, it does not need to fetch the data, which saves bus traffic compared to "Bus write" since the data itself does not need to be sent.</p>
</div>
<div class="paragraph">
<p>This is also called a Bus Upgrade message or BusUpgr, as it informs others that the value is going to be upgraded.</p>
</div>
</li>
<li>
<p>"Write back": send the data on the bus and tell someone to pick it up: either DRAM or another cache</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>When a message is sent to the bus:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>all other caches and the DRAM will see it, this is called "snooping"</p>
</li>
<li>
<p>either caches or DRAM can reply if a reply is needed, but other caches get priority to reply earlier if they can, e.g. to serve a cache request from other caches rather than going all the way to DRAM</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>When a cache receives a message, it do one or both of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>change to another MSI state</p>
</li>
<li>
<p>send a message to the bus</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>And finally, the transitions are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Modified:</p>
<div class="ulist">
<ul>
<li>
<p>"Local read": don&#8217;t need to do anything because only the current cache holds the data</p>
</li>
<li>
<p>"Local write": don&#8217;t need to do anything because only the current cache holds the data</p>
</li>
<li>
<p>"Evict": have to save data to DRAM so that our local modifications won&#8217;t be lost</p>
<div class="ulist">
<ul>
<li>
<p>Move to: Invalid</p>
</li>
<li>
<p>Send message: "Write back"</p>
</li>
</ul>
</div>
</li>
<li>
<p>"Bus read": another cache is trying to read the address which we owned exclusively.</p>
<div class="paragraph">
<p>Since we know what the latest data is, we can move to "Shared" rather than "Invalid" to possibly save time on future reads.</p>
</div>
<div class="paragraph">
<p>But to do that, we need to write the data back to DRAM to maintain the shared state consistent. The <a href="#mesi-cache-coherence-protocol">MESI cache coherence protocol</a> prevents that extra read in some cases.</p>
</div>
<div class="paragraph">
<p>And it has to be either: before the other cache gets its data from DRAM, or better, the other cache can get its data from our write back itself just like the DRAM.</p>
</div>
<div class="ulist">
<ul>
<li>
<p>Move to: Shared</p>
</li>
<li>
<p>Send message: "Write back"</p>
</li>
</ul>
</div>
</li>
<li>
<p>"Bus write": someone else will write to our address.</p>
<div class="paragraph">
<p>We don&#8217;t know what they will write, so the best bet is to move to invalid.</p>
</div>
<div class="paragraph">
<p>Since the writer will become the new sole data owner, the writer can get the cache from us without going to DRAM at all! This is fine, because the writer will be the new sole owner of the line, so DRAM can remain dirty without problems.</p>
</div>
<div class="paragraph">
<p>TODO Wikipedia requires a Flush there, why? <a href="https://electronics.stackexchange.com/questions/484830/why-is-a-flush-needed-in-the-msi-cache-coherency-protocol-when-moving-from-modif" class="bare">https://electronics.stackexchange.com/questions/484830/why-is-a-flush-needed-in-the-msi-cache-coherency-protocol-when-moving-from-modif</a></p>
</div>
<div class="ulist">
<ul>
<li>
<p>Move to: Invalid</p>
</li>
<li>
<p>Send message: "Write back"</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
</li>
<li>
<p>Shared: TODO</p>
<div class="ulist">
<ul>
<li>
<p>"Local read":</p>
</li>
<li>
<p>"Local write":</p>
</li>
<li>
<p>"Evict":</p>
</li>
<li>
<p>"Bus read":</p>
</li>
<li>
<p>"Bus write":</p>
</li>
</ul>
</div>
</li>
<li>
<p>Invalid: TODO</p>
<div class="ulist">
<ul>
<li>
<p>"Local read":</p>
</li>
<li>
<p>"Local write":</p>
</li>
<li>
<p>"Evict":</p>
</li>
<li>
<p>"Bus read":</p>
</li>
<li>
<p>"Bus write":</p>
</li>
</ul>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>TODO gem5 concrete example.</p>
</div>
<div class="sect5">
<h6 id="msi-cache-coherence-protocol-with-transient-states"><a class="anchor" href="#msi-cache-coherence-protocol-with-transient-states"></a><a class="link" href="#msi-cache-coherence-protocol-with-transient-states">37.6.1.4.1. MSI cache coherence protocol with transient states</a></h6>
<div class="paragraph">
<p>TODO understand well why those are needed.</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://learning.gem5.org/book/part3/MSI/directory.html" class="bare">http://learning.gem5.org/book/part3/MSI/directory.html</a></p>
</li>
<li>
<p><a href="https://www.researchgate.net/figure/MSI-Protocol-with-Transient-States-Adapted-from-30_fig3_2531432" class="bare">https://www.researchgate.net/figure/MSI-Protocol-with-Transient-States-Adapted-from-30_fig3_2531432</a></p>
</li>
<li>
<p><a href="http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf" class="bare">http://csg.csail.mit.edu/6.823S16/lectures/L15.pdf</a> page 28</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect4">
<h5 id="mesi-cache-coherence-protocol"><a class="anchor" href="#mesi-cache-coherence-protocol"></a><a class="link" href="#mesi-cache-coherence-protocol">37.6.1.5. MESI cache coherence protocol</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/MESI_protocol" class="bare">https://en.wikipedia.org/wiki/MESI_protocol</a></p>
</div>
<div class="paragraph">
<p>Splits the Shared of <a href="#msi-cache-coherence-protocol">MSI cache coherence protocol</a> into a new Exclusive state:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>MESI Exclusive: clean but only present in one cache</p>
</li>
<li>
<p>MESI Shared: clean but present in more that one cache</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Exclusive is entered from Invalid after a PrRd, but only if the reply came from DRAM (<a href="#can-caches-snoop-data-from-other-caches">or if we snooped that no one sent the reply to DRAM for us to read it</a>)! If the reply came from another cache, we go directly to shared instead. It is this extra information that allows for the split of S.</p>
</div>
<div class="paragraph">
<p>This is why the simplified transition diagram shown in many places e.g.: <a href="https://upload.wikimedia.org/wikipedia/commons/c/c1/Diagrama_MESI.GIF" class="bare">https://upload.wikimedia.org/wikipedia/commons/c/c1/Diagrama_MESI.GIF</a> is not a proper state machine: I can go to either S or E given a PrRd.</p>
</div>
<div class="paragraph">
<p>The advantage of this over MSI is that when we move from Exclusive to Modified, no invalidate message is required, reducing bus traffic: <a href="https://en.wikipedia.org/wiki/MESI_protocol#Advantages_of_MESI_over_MSI" class="bare">https://en.wikipedia.org/wiki/MESI_protocol#Advantages_of_MESI_over_MSI</a></p>
</div>
<div class="paragraph">
<p>This is a common case on read write modify loops. On MSI, it would:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>first do PrRd</p>
</li>
<li>
<p>send BusRd (to move any M to S), get data, and go to Shared</p>
</li>
<li>
<p>then PrWr must send BusUpgr to invalidate other Shared and move to M</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>With MESI:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the PrRd could go to E instead of S depending on who services it</p>
</li>
<li>
<p>if it does go to E, then the PrWr only moves it to M, there is no need to send BusUpgr because we know that no one else is in S</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>gem5 12c917de54145d2d50260035ba7fa614e25317a3 has two <a href="#gem5-ruby-build">Ruby</a> MESI models implemented: <code>MESI_Two_Level</code> and <code>MESI_Three_Level</code>.</p>
</div>
</div>
<div class="sect4">
<h5 id="mosi-cache-coherence-protocol"><a class="anchor" href="#mosi-cache-coherence-protocol"></a><a class="link" href="#mosi-cache-coherence-protocol">37.6.1.6. MOSI cache coherence protocol</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/MOSI_protocol" class="bare">https://en.wikipedia.org/wiki/MOSI_protocol</a> The critical MSI vs MOSI section was a bit bogus though: <a href="https://en.wikipedia.org/w/index.php?title=MOSI_protocol&amp;oldid=895443023" class="bare">https://en.wikipedia.org/w/index.php?title=MOSI_protocol&amp;oldid=895443023</a> but I edited it :-)</p>
</div>
<div class="paragraph">
<p>In MSI, it feels wasteful that an MS transaction needs to flush to memory: why do we need to flush right now, since even more caches now have that data? Why not wait until later ant try to gain something from this deferral?</p>
</div>
<div class="paragraph">
<p>The problem with doing that in MSI, is that not flushing on an MS transaction would force us to every S eviction. So we would end up flushing even after reads!</p>
</div>
<div class="paragraph">
<p>MOSI solves that by making M move to O instead of S on BusRd. Now, O is the only responsible for the flush back on eviction.</p>
</div>
<div class="paragraph">
<p>So, in case we had:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>processor 1: M</p>
</li>
<li>
<p>processor 2: I then read</p>
</li>
<li>
<p>processor 1: write</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>An MSI cache 1 would do:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>write to main memory, go to S</p>
</li>
<li>
<p>BusUpgr, go back to M, 2 back to I</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>and MOSI would do:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>go to O (no bus traffic)</p>
</li>
<li>
<p>BusUpgr, go back to M</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>This therefore saves one memory write through and its bus traffic.</p>
</div>
</div>
<div class="sect4">
<h5 id="moesi"><a class="anchor" href="#moesi"></a><a class="link" href="#moesi">37.6.1.7. MOESI cache coherence protocol</a></h5>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/MOESI_protocol" class="bare">https://en.wikipedia.org/wiki/MOESI_protocol</a></p>
</div>
<div class="paragraph">
<p><a href="#mesi-cache-coherence-protocol">MESI cache coherence protocol</a> + <a href="#mosi-cache-coherence-protocol">MOSI cache coherence protocol</a>, not much else to it!</p>
</div>
<div class="paragraph">
<p>In gem5 9fc9c67b4242c03f165951775be5cd0812f2a705, MOESI is the default cache coherency protocol of the <a href="#gem5-ruby-build">classic memory system</a> as shown at <a href="#what-is-the-coherency-protocol-implemented-by-the-classic-cache-system-in-gem5">Section 24.22.4.3.1, &#8220;What is the coherency protocol implemented by the classic cache system in gem5?&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>A good an simple example showing several MOESI transitions in the classic memory model can be seen at: <a href="#gem5-event-queue-atomicsimplecpu-syscall-emulation-freestanding-example-analysis-with-caches-and-multiple-cpus">Section 24.22.4.4, &#8220;gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis with caches and multiple CPUs&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>gem5 12c917de54145d2d50260035ba7fa614e25317a3 has several <a href="#gem5-ruby-build">Ruby</a> MOESI models implemented: <code>MOESI_AMD_Base</code>, <code>MOESI_CMP_directory</code>, <code>MOESI_CMP_token</code> and <code>MOESI_hammer</code>.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="about-this-repo"><a class="anchor" href="#about-this-repo"></a><a class="link" href="#about-this-repo">38. About this repo</a></h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="supported-hosts"><a class="anchor" href="#supported-hosts"></a><a class="link" href="#supported-hosts">38.1. Supported hosts</a></h3>
<div class="paragraph">
<p>The host requirements depend a lot on which examples you want to run.</p>
</div>
<div class="paragraph">
<p>Some setups of this repository are very portable, notably setups under <a href="#userland-setup">Userland setup</a>, e.g. <a href="#c">C</a>, and will likely work on any host system with minimal modification.</p>
</div>
<div class="paragraph">
<p>The least portable setups are those that require Buildroot and crosstool-NG.</p>
</div>
<div class="paragraph">
<p>We tend to test this repo the most on the latest Ubuntu and on the latest <a href="https://askubuntu.com/questions/16366/whats-the-difference-between-a-long-term-support-release-and-a-normal-release">Ubuntu LTS</a>.</p>
</div>
<div class="paragraph">
<p>For other Linux distros, everything will likely also just work if you install the analogous required packages for your distro.</p>
</div>
<div class="paragraph">
<p>Find out the packages that we install with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cat ./setup
./build --download-dependencies --dry-run &lt;some-target&gt; | less</pre>
</div>
</div>
<div class="paragraph">
<p>and then just look for the <code>apt-get</code> commands shown on the log.</p>
</div>
<div class="paragraph">
<p>After installing the missing packages for your distro, do the build with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --download-dependencies --no-apt &lt;some-target&gt;</pre>
</div>
</div>
<div class="paragraph">
<p>which does everything as normal, except that it skips any <code>apt</code> commands.</p>
</div>
<div class="paragraph">
<p>If something does not work however, <a href="#docker">Docker host setup</a> should just work on any Linux distro.</p>
</div>
<div class="paragraph">
<p>Native Windows is unlikely feasible for Buildroot setups because Buildroot is a huge set of GNU Make scripts + host tools, just do everything from inside an Ubuntu in VirtualBox instance in that case.</p>
</div>
<div class="paragraph">
<p>Pull requests with ports to new host systems and reports on issues that things work or don&#8217;t work on your host are welcome.</p>
</div>
</div>
<div class="sect2">
<h3 id="common-build-issues"><a class="anchor" href="#common-build-issues"></a><a class="link" href="#common-build-issues">38.2. Common build issues</a></h3>
<div class="sect3">
<h4 id="put-source-uris-in-sources"><a class="anchor" href="#put-source-uris-in-sources"></a><a class="link" href="#put-source-uris-in-sources">38.2.1. You must put some 'source' URIs in your sources.list</a></h4>
<div class="paragraph">
<p>If <code>./build --download-dependencies</code> fails with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>E: You must put some 'source' URIs in your sources.list</pre>
</div>
</div>
<div class="paragraph">
<p>see this: <a href="https://askubuntu.com/questions/496549/error-you-must-put-some-source-uris-in-your-sources-list/857433#857433" class="bare">https://askubuntu.com/questions/496549/error-you-must-put-some-source-uris-in-your-sources-list/857433#857433</a> I don&#8217;t know how to automate this step. Why, Ubuntu, why.</p>
</div>
</div>
<div class="sect3">
<h4 id="build-from-downloaded-source-zip-files"><a class="anchor" href="#build-from-downloaded-source-zip-files"></a><a class="link" href="#build-from-downloaded-source-zip-files">38.2.2. Build from downloaded source zip files</a></h4>
<div class="paragraph">
<p>It does not work if you just download the <code>.zip</code> with the sources for this repository from GitHub because we use <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/.gitmodules">Git submodules</a>, you must clone this repo.</p>
</div>
<div class="paragraph">
<p><code>./build --download-dependencies</code> then fetches only the required submodules for you.</p>
</div>
</div>
</div>
<div class="sect2">
<h3 id="run-command-after-boot"><a class="anchor" href="#run-command-after-boot"></a><a class="link" href="#run-command-after-boot">38.3. Run command after boot</a></h3>
<div class="paragraph">
<p>If you just want to run a command after boot ends without thinking much about it, just use the <code>--eval-after</code> option, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after 'echo hello'</pre>
</div>
</div>
<div class="paragraph">
<p>This option passes the command to our init scripts through <a href="#kernel-command-line-parameters">Kernel command line parameters</a>, and uses a few clever tricks along the way to make it just work.</p>
</div>
<div class="paragraph">
<p>See <a href="#init">init</a> for the gory details.</p>
</div>
</div>
<div class="sect2">
<h3 id="default-command-line-arguments"><a class="anchor" href="#default-command-line-arguments"></a><a class="link" href="#default-command-line-arguments">38.4. Default command line arguments</a></h3>
<div class="paragraph">
<p>It gets annoying to retype <code>--arch aarch64</code> for every single command, or to remember <code>--config</code> setups.</p>
</div>
<div class="paragraph">
<p>So simplify that, do:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cp config.py data/</pre>
</div>
</div>
<div class="paragraph">
<p>and then edit the <code>data/config</code> file to your needs.</p>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/config.py">config.py</a></p>
</div>
<div class="paragraph">
<p>You can also choose a different configuration file explicitly with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --config data/config2.py</pre>
</div>
</div>
<div class="paragraph">
<p>Almost all options names are automatically deduced from their command line <code>--help</code> name: just replace <code>-</code> with <code>_</code>.</p>
</div>
<div class="paragraph">
<p>More precisely, we use the <code>dest=</code> value of Python&#8217;s <a href="https://docs.python.org/3/library/argparse.html">argparse module</a>.</p>
</div>
<div class="paragraph">
<p>To get a list of all global options that you can use, try:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar --type input</pre>
</div>
</div>
<div class="paragraph">
<p>but note that this does not include script specific options.</p>
</div>
</div>
<div class="sect2">
<h3 id="documentation"><a class="anchor" href="#documentation"></a><a class="link" href="#documentation">38.5. Documentation</a></h3>
<div class="paragraph">
<p>To learn how to build the documentation see: <a href="#build-the-documentation">Section 2.10, &#8220;Build the documentation&#8221;</a>.</p>
</div>
<div class="sect3">
<h4 id="documentation-verification"><a class="anchor" href="#documentation-verification"></a><a class="link" href="#documentation-verification">38.5.1. Documentation verification</a></h4>
<div class="paragraph">
<p>When running <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-doc">build-doc</a>, we do the following checks:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>&lt;&lt;&gt;&gt;</code> inner links are not broken</p>
</li>
<li>
<p><code>link:somefile[]</code> links point to paths that exist via <a href="#asciidoctor-extract-link-targets">asciidoctor/extract-link-targets</a>. Upstream wontfix at: <a href="https://github.com/asciidoctor/asciidoctor/issues/3210" class="bare">https://github.com/asciidoctor/asciidoctor/issues/3210</a></p>
</li>
<li>
<p>all links in non-README files to README IDs exist via <code>git grep</code> + <a href="#asciidoctor-extract-header-ids">asciidoctor/extract-header-ids</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The scripts prints what you have to fix and exits with an error status if there are any errors.</p>
</div>
<div class="sect4">
<h5 id="asciidoctor-extract-link-targets"><a class="anchor" href="#asciidoctor-extract-link-targets"></a><a class="link" href="#asciidoctor-extract-link-targets">38.5.1.1. asciidoctor/extract-link-targets</a></h5>
<div class="paragraph">
<p>Documentation for <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/asciidoctor/extract-link-targets">asciidoctor/extract-link-targets</a></p>
</div>
<div class="paragraph">
<p>Extract link targets from Asciidoctor document.</p>
</div>
<div class="paragraph">
<p>Usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./asciidoctor/extract-link-targets README.adoc</pre>
</div>
</div>
<div class="paragraph">
<p>Output: one link target per line.</p>
</div>
<div class="paragraph">
<p>Hastily hacked from: <a href="https://asciidoctor.org/docs/user-manual/#inline-macro-processor-example" class="bare">https://asciidoctor.org/docs/user-manual/#inline-macro-processor-example</a></p>
</div>
</div>
<div class="sect4">
<h5 id="asciidoctor-extract-header-ids"><a class="anchor" href="#asciidoctor-extract-header-ids"></a><a class="link" href="#asciidoctor-extract-header-ids">38.5.1.2. asciidoctor/extract-header-ids</a></h5>
<div class="paragraph">
<p>Documentation for <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/asciidoctor/extract-header-ids">asciidoctor/extract-header-ids</a></p>
</div>
<div class="paragraph">
<p>Extract header IDs, both auto-generated and manually given.</p>
</div>
<div class="paragraph">
<p>E.g., for the document <code>test.adoc</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>= Auto generated

[[explicitly-given]]
== La la</pre>
</div>
</div>
<div class="paragraph">
<p>the script:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./asciidoctor/extract-header-ids test.adoc</pre>
</div>
</div>
<div class="paragraph">
<p>produces:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>auto-generated
explicitly-given</pre>
</div>
</div>
<div class="paragraph">
<p>One application we have in mind for this is that as of 2.0.10 Asciidoctor does not warn on header ID collisions between auto-generated IDs: <a href="https://github.com/asciidoctor/asciidoctor/issues/3147" class="bare">https://github.com/asciidoctor/asciidoctor/issues/3147</a> But this script doesn&#8217;t solve that yet as it would require generating the section IDs without the <code>-N</code> suffix. Section generation happens at <code>Section.generate_id</code> in Asciidoctor code.</p>
</div>
<div class="paragraph">
<p>Hastily hacked from: <a href="https://asciidoctor.org/docs/user-manual/#https://asciidoctor.org/docs/user-manual/#tree-processor-example" class="bare">https://asciidoctor.org/docs/user-manual/#https://asciidoctor.org/docs/user-manual/#tree-processor-example</a> until I noticed that that example had a bug at the time and so fixed it here: <a href="https://github.com/asciidoctor/asciidoctor/issues/3363" class="bare">https://github.com/asciidoctor/asciidoctor/issues/3363</a></p>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="asciidoctor-link-target-up-rb"><a class="anchor" href="#asciidoctor-link-target-up-rb"></a><a class="link" href="#asciidoctor-link-target-up-rb">38.6. asciidoctor/link-target-up.rb</a></h3>
<div class="paragraph">
<p>The Asciidoctor extension scripts:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>link:asciidoctor-link-up.rb</p>
</li>
<li>
<p>link:asciidoctor-link-github.rb</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>hack the README <code>link:</code> targets to make them work from:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>inside the <code>out/</code> directory with <code>../</code></p>
</li>
<li>
<p><a href="#github-pages">GitHub pages</a>, with explicit GitHub blob URLs</p>
</li>
</ul>
</div>
<div class="sect3">
<h4 id="github-pages"><a class="anchor" href="#github-pages"></a><a class="link" href="#github-pages">38.6.1. GitHub pages</a></h4>
<div class="paragraph">
<p>As mentioned before the TOC, we have to push this README to GitHub pages due to: <a href="https://github.com/isaacs/github/issues/1610" class="bare">https://github.com/isaacs/github/issues/1610</a></p>
</div>
<div class="paragraph">
<p>For now, instead of pushing with <code>git push</code>, I just remember to always push with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./publish-gh-pages</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/publish-gh-pages">publish-gh-pages</a></p>
</div>
<div class="paragraph">
<p>I&#8217;m going this way for now because:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>the Jekyll Asciidoctor plugin is not enabled by default on GitHub: <a href="https://webapps.stackexchange.com/questions/114606/can-github-pages-render-asciidoc" class="bare">https://webapps.stackexchange.com/questions/114606/can-github-pages-render-asciidoc</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/1797074/local-executing-hook-after-a-git-push">post-push hooks don&#8217;t exist</a></p>
</li>
<li>
<p>I&#8217;m lazy to setup a proper Travis CI push</p>
</li>
<li>
<p>I&#8217;m the only contributor essentially, so no problems with pull requests</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The only files used by the GitHub pages are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/Gemfile">Gemfile</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/_config.yml">_config.yml</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="clean-the-build"><a class="anchor" href="#clean-the-build"></a><a class="link" href="#clean-the-build">38.7. Clean the build</a></h3>
<div class="paragraph">
<p>You did something crazy, and nothing seems to work anymore?</p>
</div>
<div class="paragraph">
<p>All our build outputs are stored under <code>out/</code>, so the coarsest and most effective thing you can do is:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>rm -rf out</pre>
</div>
</div>
<div class="paragraph">
<p>This implies a full rebuild for all archs however, so you might first want to explore finer grained cleans first.</p>
</div>
<div class="paragraph">
<p>All our individual <code>build-*</code> scripts have a <code>--clean</code> option to completely nuke their builds:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --clean
./build-qemu --clean
./build-buildroot --clean</pre>
</div>
</div>
<div class="paragraph">
<p>Verify with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls "$(./getvar qemu_build_dir)"
ls "$(./getvar gem5_build_dir)"
ls "$(./getvar buildroot_build_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>Note that host tools like QEMU and gem5 store all archs in a single directory to factor out build objects, so cleaning one arch will clean all of them.</p>
</div>
<div class="paragraph">
<p>To only nuke only one Buildroot package, we can use the <a href="https://buildroot.org/downloads/manual/manual.html#pkg-build-steps"><code>-dirclean</code></a> Buildroot target:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --no-all -- &lt;package-name&gt;-dirclean</pre>
</div>
</div>
<div class="paragraph">
<p>e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --no-all -- sample_package-dirclean</pre>
</div>
</div>
<div class="paragraph">
<p>Verify with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls "$(./getvar buildroot_build_build_dir)"</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="custom-build-directory"><a class="anchor" href="#custom-build-directory"></a><a class="link" href="#custom-build-directory">38.8. Custom build directory</a></h3>
<div class="paragraph">
<p>For now there is no way to change the build directory from <code>out/</code> (resp. <code>out.docker</code> for &lt;&lt;docker&gt;.) to something else.</p>
</div>
<div class="paragraph">
<p>However, if you just want to place the build storage in your hard drive and the source in your SSD, which is a good configuration if you are doing lots of builds, just create a symlink as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mkdir -p /mnt/hd/linux-kernel-module-cheat-out
ln -s out /mnt/hd/linux-kernel-module-cheat-out</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="ccache"><a class="anchor" href="#ccache"></a><a class="link" href="#ccache">38.9. ccache</a></h3>
<div class="paragraph">
<p><a href="https://en.wikipedia.org/wiki/Ccache">ccache</a> <a href="#benchmark-builds">might</a> save you a lot of re-build when you decide to <a href="#clean-the-build">Clean the build</a> or create a new <a href="#build-variants">build variant</a>.</p>
</div>
<div class="paragraph">
<p>We have ccache enabled for everything we build by default.</p>
</div>
<div class="paragraph">
<p>However, you likely want to add the following to your <code>.bashrc</code> to take better advantage of <code>ccache</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>export CCACHE_DIR=~/.ccache
export CCACHE_MAXSIZE="20G"</pre>
</div>
</div>
<div class="paragraph">
<p>We cannot automate this because you have to decide:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>should I store my cache on my HD or SSD?</p>
</li>
<li>
<p>how big is my build, and how many build configurations do I need to keep around at a time?</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If you don&#8217;t those variables it, the default is to use <code>~/.buildroot-ccache</code> with <code>5G</code>, which is a bit small for us.</p>
</div>
<div class="paragraph">
<p>To check if <code>ccache</code> is working, run this command while a build is running on another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>watch -n1 'make -C "$(./getvar buildroot_build_dir)" ccache-stats'</pre>
</div>
</div>
<div class="paragraph">
<p>or if you have it installed on host and the environment variables exported simply with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>watch -n1 'ccache -s'</pre>
</div>
</div>
<div class="paragraph">
<p>and then watch the miss or hit counts go up.</p>
</div>
<div class="paragraph">
<p>We have <a href="https://buildroot.org/downloads/manual/manual.html#ccache">enabled ccached</a> builds by default.</p>
</div>
<div class="paragraph">
<p><code>BR2_CCACHE_USE_BASEDIR=n</code> is used for Buildroot, which means that:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>absolute paths are used and GDB can find source files</p>
</li>
<li>
<p>but builds are not reused across separated LKMC directories</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>ccache can be disabled with the <code>--no-ccache</code> option as in:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-gem5 --no-ccache</pre>
</div>
</div>
<div class="paragraph">
<p>This can be useful to <a href="#benchmark-builds">benchmark builds</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="getvar"><a class="anchor" href="#getvar"></a><a class="link" href="#getvar">38.10. getvar</a></h3>
<div class="paragraph">
<p>The <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/getvar">getvar</a> helper script can print the values of internal LKMC variables.</p>
</div>
<div class="paragraph">
<p>Within our Python scripts such as <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/common.py">common.py</a>, those variable are visible as <code>self.env[&lt;var&gt;]</code>.</p>
</div>
<div class="paragraph">
<p>For example, to find the Buildroot output directory for an <code>aarch64</code> build, you could use:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar --arch aarch64 buildroot_build_dir</pre>
</div>
</div>
<div class="paragraph">
<p>which as of LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7 outputs:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/path/to/linux-kernel-module-cheat/out/buildroot/build/default/aarch64</pre>
</div>
</div>
<div class="paragraph">
<p>You can also list all available variables in one go with just:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar</pre>
</div>
</div>
<div class="paragraph">
<p>Using <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/getvar">getvar</a> makes it possible to make Bash scripts more portable if for example directory structure changes across LKMC versions.</p>
</div>
<div class="paragraph">
<p>For this reason, we use it in particular often in this README to reduce the need for refactoring.</p>
</div>
<div class="sect3">
<h4 id="run-toolchain"><a class="anchor" href="#run-toolchain"></a><a class="link" href="#run-toolchain">38.10.1. run-toolchain</a></h4>
<div class="paragraph">
<p>While you could just manually find/learn the path to toolchain tools, e.g. in LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7 they are:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-gcc userland/c/hello.c
./out/buildroot/build/default/aarch64/host/bin/aarch64-buildroot-linux-gnu-objdump -D a.out</pre>
</div>
</div>
<div class="paragraph">
<p>you can save some typing and get portability across directory structure changes with our <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/run-toolchain">run-toolchain</a> helper:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch aarch64 gcc -- userland/c/hello.c
./run-toolchain --arch aarch64 objdump -- -D a.out</pre>
</div>
</div>
<div class="paragraph">
<p>This plays nicely with <a href="#getvar">getvar</a> e.g. you could disassembly <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/hello.c">userland/c/hello.c</a> with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-toolchain --arch aarch64 objdump -- -D $(./getvar --arch aarch64 userland_build_dir)/c/hello.out</pre>
</div>
</div>
<div class="paragraph">
<p>however disassembly is such a common use case that we have a shortcut for it: <a href="#disas">disas</a>.</p>
</div>
<div class="paragraph">
<p>Alternatively, if you just need a variable to feed into your own Build system, you can also use <a href="#getvar">getvar</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar --arch aarch64 toolchain_prefix</pre>
</div>
</div>
<div class="paragraph">
<p>which outputs as of LKMC b15a0e455d691afa49f3b813ad9b09394dfb02b7:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>/path/to/linux-kernel-module-cheat/out/buildroot/build/default/aarch64/host/usr/bin/aarch64-buildroot-linux-gnu</pre>
</div>
</div>
<div class="sect4">
<h5 id="disas"><a class="anchor" href="#disas"></a><a class="link" href="#disas">38.10.1.1. disas</a></h5>
<div class="paragraph">
<p>Since disassembly of a single function of a LKMC executable with GDB is such a common use case for <a href="#run-toolchain">run-toolchain</a> via <a href="https://stackoverflow.com/questions/22769246/how-to-disassemble-one-single-function-using-objdump" class="bare">https://stackoverflow.com/questions/22769246/how-to-disassemble-one-single-function-using-objdump</a>, we have this shortcut for it.</p>
</div>
<div class="paragraph">
<p>For example to disassemle a function from an <a href="#userland-content">userland binary</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --userland userland/c/hello.c main</pre>
</div>
</div>
<div class="paragraph">
<p>or to disassemble a function from the <a href="#linux-kernel">Linux kernel</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>and a <a href="#baremetal-setup">baremetal</a> executable:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./disas --arch aarch64 --baremetal baremetal/arch/aarch64/no_bootloader/exit.S _start</pre>
</div>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="rebuild-buildroot-while-running"><a class="anchor" href="#rebuild-buildroot-while-running"></a><a class="link" href="#rebuild-buildroot-while-running">38.11. Rebuild Buildroot while running</a></h3>
<div class="paragraph">
<p>It is not possible to rebuild the root filesystem while running QEMU because QEMU holds the file qcow2 file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>error while converting qcow2: Failed to get "write" lock</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="simultaneous-runs"><a class="anchor" href="#simultaneous-runs"></a><a class="link" href="#simultaneous-runs">38.12. Simultaneous runs</a></h3>
<div class="paragraph">
<p>When doing long simulations sweeping across multiple system parameters, it becomes fundamental to do multiple simulations in parallel.</p>
</div>
<div class="paragraph">
<p>This is specially true for gem5, which runs much slower than QEMU, and cannot use multiple host cores to speed up the simulation: <a href="https://github.com/cirosantilli2/gem5-issues/issues/15" class="bare">https://github.com/cirosantilli2/gem5-issues/issues/15</a>, so the only way to parallelize is to run multiple instances in parallel.</p>
</div>
<div class="paragraph">
<p>This also has a good synergy with <a href="#build-variants">Build variants</a>.</p>
</div>
<div class="paragraph">
<p>First shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run</pre>
</div>
</div>
<div class="paragraph">
<p>Another shell:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --run-id 1</pre>
</div>
</div>
<div class="paragraph">
<p>and now you have two QEMU instances running in parallel.</p>
</div>
<div class="paragraph">
<p>The default run id is <code>0</code>.</p>
</div>
<div class="paragraph">
<p>Our scripts solve two difficulties with simultaneous runs:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>port conflicts, e.g. GDB and <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/gem5-shell">gem5-shell</a></p>
</li>
<li>
<p>output directory conflicts, e.g. traces and gem5 stats overwriting one another</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Each run gets a separate output directory. For example:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --run-id 0 &amp;&gt;/dev/null &amp;
./run --arch aarch64 --emulator gem5 --run-id 1 &amp;&gt;/dev/null &amp;</pre>
</div>
</div>
<div class="paragraph">
<p>produces two separate <a href="#m5out-directory"><code>m5out</code> directories</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo "$(./getvar --arch aarch64 --emulator gem5 --run-id 0 m5out_dir)"
echo "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 m5out_dir)"</pre>
</div>
</div>
<div class="paragraph">
<p>and the gem5 host executable stdout and stderr can be found at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>less "$(./getvar --arch aarch64 --emulator gem5 --run-id 0 termout_file)"
less "$(./getvar --arch aarch64 --emulator gem5 --run-id 1 termout_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>Each line is prepended with the timestamp in seconds since the start of the program when it appeared.</p>
</div>
<div class="paragraph">
<p>To have more semantic output directories names for later inspection, you can use a non numeric string for the run ID, and indicate the port offset explicitly:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 --emulator gem5 --run-id some-experiment --port-offset 1</pre>
</div>
</div>
<div class="paragraph">
<p><code>--port-offset</code> defaults to the run ID when that is a number.</p>
</div>
<div class="paragraph">
<p>Like <a href="#cpu-architecture">CPU architecture</a>, you will need to pass the <code>-n</code> option to anything that needs to know runtime information, e.g. <a href="#gdb">GDB step debug</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --run-id 1
./run-gdb --run-id 1</pre>
</div>
</div>
<div class="paragraph">
<p>To run multiple gem5 checkouts, see: <a href="#gem5-worktree">Section 38.13.3.1, &#8220;gem5 worktree&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>Implementation note: we create multiple namespaces for two things:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>run output directory</p>
</li>
<li>
<p>ports</p>
<div class="ulist">
<ul>
<li>
<p>QEMU allows setting all ports explicitly.</p>
<div class="paragraph">
<p>If a port is not free, it just crashes.</p>
</div>
<div class="paragraph">
<p>We assign a contiguous port range for each run ID.</p>
</div>
</li>
<li>
<p>gem5 automatically increments ports until it finds a free one.</p>
<div class="paragraph">
<p>gem5 60600f09c25255b3c8f72da7fb49100e2682093a does not seem to expose a way to set the terminal and VNC ports from <code>fs.py</code>, so we just let gem5 assign the ports itself, and use <code>-n</code> only to match what it assigned. Those ports both appear on <a href="#gem5-config-ini">gem5 config.ini</a>.</p>
</div>
<div class="paragraph">
<p>The GDB port can be assigned on <code>gem5.opt --remote-gdb-port</code>, but it does not appear on <code>config.ini</code>.</p>
</div>
</li>
</ul>
</div>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="build-variants"><a class="anchor" href="#build-variants"></a><a class="link" href="#build-variants">38.13. Build variants</a></h3>
<div class="paragraph">
<p>It often happens that you are comparing two versions of the build, a good and a bad one, and trying to figure out why the bad one is bad.</p>
</div>
<div class="paragraph">
<p>Our build variants system allows you to keep multiple built versions of all major components, so that you can easily switching between running one or the other.</p>
</div>
<div class="sect3">
<h4 id="linux-kernel-build-variants"><a class="anchor" href="#linux-kernel-build-variants"></a><a class="link" href="#linux-kernel-build-variants">38.13.1. Linux kernel build variants</a></h4>
<div class="paragraph">
<p>If you want to keep two builds around, one for the latest Linux version, and the other for Linux <code>v4.16</code>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Build master.
./build-linux

# Build another branch.
git -C "$(./getvar linux_source_dir)" fetch --tags --unshallow
git -C "$(./getvar linux_source_dir)" checkout v4.16
./build-linux --linux-build-id v4.16

# Restore master.
git -C "$(./getvar linux_source_dir)" checkout -

# Run master.
./run

# Run another branch.
./run --linux-build-id v4.16</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>git fetch --unshallow</code> is needed the first time because <code>./build --download-dependencies</code> only does a shallow clone of the Linux kernel to save space and time, see also: <a href="https://stackoverflow.com/questions/6802145/how-to-convert-a-git-shallow-clone-to-a-full-clone" class="bare">https://stackoverflow.com/questions/6802145/how-to-convert-a-git-shallow-clone-to-a-full-clone</a></p>
</div>
<div class="paragraph">
<p>The <code>--linux-build-id</code> option should be passed to all scripts that support it, much like <code>--arch</code> for the <a href="#cpu-architecture">CPU architecture</a>, e.g. to step debug:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --linux-build-id v4.16</pre>
</div>
</div>
<div class="paragraph">
<p>To run both kernels simultaneously, one on each QEMU instance, see: <a href="#simultaneous-runs">Section 38.12, &#8220;Simultaneous runs&#8221;</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="qemu-build-variants"><a class="anchor" href="#qemu-build-variants"></a><a class="link" href="#qemu-build-variants">38.13.2. QEMU build variants</a></h4>
<div class="paragraph">
<p>Analogous to the <a href="#linux-kernel-build-variants">Linux kernel build variants</a> but with the <code>--qemu-build-id</code> option instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-qemu
git -C "$(./getvar qemu_source_dir)" checkout v2.12.0
./build-qemu --qemu-build-id v2.12.0
git -C "$(./getvar qemu_source_dir)" checkout -
./run
./run --qemu-build-id v2.12.0</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="gem5-build-variants"><a class="anchor" href="#gem5-build-variants"></a><a class="link" href="#gem5-build-variants">38.13.3. gem5 build variants</a></h4>
<div class="paragraph">
<p>Analogous to the <a href="#linux-kernel-build-variants">Linux kernel build variants</a> but with the <code>--gem5-build-id</code> option instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Build master.
./build-gem5

# Build another branch.
git -C "$(./getvar gem5_source_dir)" checkout some-branch
./build-gem5 --gem5-build-id some-branch

# Restore master.
git -C "$(./getvar gem5_source_dir)" checkout -

# Run master.
./run --emulator gem5

# Run another branch.
git -C "$(./getvar gem5_source_dir)" checkout some-branch
./run --gem5-build-id some-branch --emulator gem5</pre>
</div>
</div>
<div class="paragraph">
<p>Don&#8217;t forget however that gem5 has Python scripts in its source code tree, and that those must match the source code of a given build.</p>
</div>
<div class="paragraph">
<p>Therefore, you can&#8217;t forget to checkout to the sources to that of the corresponding build before running, unless you explicitly tell gem5 to use a non-default source tree with <a href="#gem5-worktree">gem5 worktree</a>. This becomes inevitable when you want to launch multiple simultaneous runs at different checkouts.</p>
</div>
<div class="sect4">
<h5 id="gem5-worktree"><a class="anchor" href="#gem5-worktree"></a><a class="link" href="#gem5-worktree">38.13.3.1. gem5 worktree</a></h5>
<div class="paragraph">
<p><a href="#gem5-build-variants"><code>--gem5-build-id</code></a> goes a long way, but if you want to seamlessly switch between two gem5 tress without checking out multiple times, then <code>--gem5-worktree</code> is for you.</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Build gem5 at the revision in the gem5 submodule.
./build-gem5

# Create a branch at the same revision as the gem5 submodule.
./build-gem5 --gem5-worktree my-new-feature
cd "$(./getvar --gem5-worktree my-new-feature)"
vim create-bugs
git add .
git commit -m 'Created a bug'
cd -
./build-gem5 --gem5-worktree my-new-feature

# Run the submodule.
./run --emulator gem5 --run-id 0 &amp;&gt;/dev/null &amp;

# Run the branch the need to check out anything.
# With --gem5-worktree, we can do both runs at the same time!
./run --emulator gem5 --gem5-worktree my-new-feature --run-id 1 &amp;&gt;/dev/null &amp;</pre>
</div>
</div>
<div class="paragraph">
<p><code>--gem5-worktree &lt;worktree-id&gt;</code> automatically creates:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a <a href="https://git-scm.com/docs/git-worktree">Git worktree</a> of gem5 if one didn&#8217;t exit yet for <code>&lt;worktree-id&gt;</code></p>
</li>
<li>
<p>a separate build directory, exactly like <code>--gem5-build-id my-new-feature</code> would</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We promise that the scripts sill never touch that worktree again once it has been created: it is now up to you to manage the code manually.</p>
</div>
<div class="paragraph">
<p><code>--gem5-worktree</code> is required if you want to do multiple simultaneous runs of different gem5 versions, because each gem5 build needs to use the matching Python scripts inside the source tree.</p>
</div>
<div class="paragraph">
<p>The difference between <code>--gem5-build-id</code> and <code>--gem5-worktree</code> is that <code>--gem5-build-id</code> specifies only the gem5 build output directory, while <code>--gem5-worktree</code> specifies the source input directory.</p>
</div>
<div class="paragraph">
<p>Each Git worktree needs a branch name, and we append the <code>wt/</code> prefix to the <code>--gem5-worktree</code> value, where <code>wt</code> stands for <code>WorkTree</code>. This is done to allow us to checkout to a test <code>some-branch</code> branch under <code>submodules/gem5</code> and still use <code>--gem5-worktree some-branch</code>, without conflict for the worktree branch, which can only be checked out once.</p>
</div>
</div>
<div class="sect4">
<h5 id="gem5-private-source-trees"><a class="anchor" href="#gem5-private-source-trees"></a><a class="link" href="#gem5-private-source-trees">38.13.3.2. gem5 private source trees</a></h5>
<div class="paragraph">
<p>Suppose that you are working on a private fork of gem5, but you want to use this repository to develop it as well.</p>
</div>
<div class="paragraph">
<p>Simply adding your private repository as a remote to <code>submodules/gem5</code> is dangerous, as you might forget and push your private work by mistake one day.</p>
</div>
<div class="paragraph">
<p>Even removing remotes is not safe enough, since <code>git submodule update</code> and other submodule commands can restore the old public remote.</p>
</div>
<div class="paragraph">
<p>Instead, we provide the following safer process.</p>
</div>
<div class="paragraph">
<p>First do a separate private clone of you private repository outside of this repository:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git clone https://my.private.repo.com/my-fork/gem5.git gem5-internal
gem5_internal="$(pwd)/gem5-internal"</pre>
</div>
</div>
<div class="paragraph">
<p>Next, when you want to build with the private repository, use the <code>--gem5-build-dir</code> and <code>--gem5-source-dir</code> argument to override our default gem5 source and build locations:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd linux-kernel-module-cheat
./build-gem5 \
  --gem5-build-dir "${gem5_internal}/build" \
  --gem5-source-dir "$gem5_internal" \
;
./run-gem5 \
  --gem5-build-dir "${gem5_internal}/build" \
  --gem5-source-dir "$gem5_internal" \
;</pre>
</div>
</div>
<div class="paragraph">
<p>With this setup, both your private gem5 source and build are safely kept outside of this public repository.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="buildroot-build-variants"><a class="anchor" href="#buildroot-build-variants"></a><a class="link" href="#buildroot-build-variants">38.13.4. Buildroot build variants</a></h4>
<div class="paragraph">
<p>Allows you to have multiple versions of the GCC toolchain or root filesystem.</p>
</div>
<div class="paragraph">
<p>Analogous to the <a href="#linux-kernel-build-variants">Linux kernel build variants</a> but with the <code>--build-id</code> option instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot
git -C "$(./getvar buildroot_source_dir)" checkout 2018.05
./build-buildroot --buildroot-build-id 2018.05
git -C "$(./getvar buildroot_source_dir)" checkout -
./run
./run --buildroot-build-id 2018.05</pre>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="optimization-level-of-a-build"><a class="anchor" href="#optimization-level-of-a-build"></a><a class="link" href="#optimization-level-of-a-build">38.14. Optimization level of a build</a></h3>
<div class="paragraph">
<p>The <code>--optimization-level</code> option is available on all build scripts and sets the given GCC `-`O optimization level where it has been implemented for guest binaries.</p>
</div>
<div class="paragraph">
<p>The default optimization level is <code>-O0</code> to improve guest visibility.</p>
</div>
<div class="paragraph">
<p>To keep things sane, you generally want to create a separate <a href="#build-variants">build variant</a> for each optimization level, e.g. to create an <code>-O3</code> build:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-userland --optimization-level 3 --userland-build-id o3
./run --userland userland/c/hello.c --userland-build-id o3</pre>
</div>
</div>
<div class="paragraph">
<p>Note that for some guest content, there are hard technical challenges why we are not able to forward <code>-O</code>, notably the linux kernel: <a href="#kernel-o0">Disable kernel compiler optimizations</a>.</p>
</div>
<div class="paragraph">
<p>Our emulators however are build with higher optimization levels by default otherwise running anything would be too unbearably slow.</p>
</div>
<div class="paragraph">
<p>Emulator builds are also controlled with other mechanisms instead of <code>--optimization-level</code> as explained at: <a href="#debug-the-emulator">Debug the emulator</a>.</p>
</div>
</div>
<div class="sect2">
<h3 id="directory-structure"><a class="anchor" href="#directory-structure"></a><a class="link" href="#directory-structure">38.15. Directory structure</a></h3>
<div class="sect3">
<h4 id="lkmc-directory"><a class="anchor" href="#lkmc-directory"></a><a class="link" href="#lkmc-directory">38.15.1. lkmc directory</a></h4>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc/">lkmc/</a> contains sources and headers that are shared across kernel modules, userland and baremetal examples.</p>
</div>
<div class="paragraph">
<p>We chose this awkward name so that our includes will have an <code>lkmc/</code> prefix.</p>
</div>
<div class="paragraph">
<p>Another option would have been to name it as <code>includes/lkmc</code>, but that would make paths longer, and we might want to store source code in that directory as well in the future.</p>
</div>
<div class="sect4">
<h5 id="userland-objects-vs-header-only"><a class="anchor" href="#userland-objects-vs-header-only"></a><a class="link" href="#userland-objects-vs-header-only">38.15.1.1. Userland objects vs header-only</a></h5>
<div class="paragraph">
<p>When factoring out functionality across userland examples, there are two main options:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>use header-only implementations</p>
</li>
<li>
<p>use separate C files and link to separate objects.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The downsides of the header-only implementation are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>slower compilation time, especially for C++</p>
</li>
<li>
<p>cannot call C implementations from assembly files</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The advantages of header-only implementations are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>easier to use, just <code>#include</code> and you are done, no need to modify build metadata.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>As a result, we are currently using the following rule:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>if something is only going to be used from C and not assembly, define it in a header which is easier to use</p>
<div class="paragraph">
<p>The slower compilation should be OK as long as split functionality amongst different headers and only include the required ones.</p>
</div>
<div class="paragraph">
<p>Also we don&#8217;t have a choice in the case of C++ template, which must stay in headers.</p>
</div>
</li>
<li>
<p>if the functionality will be called from assembly, then we don&#8217;t have a choice, and must add it to a separate source file and link against it.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect3">
<h4 id="buildroot-packages-directory"><a class="anchor" href="#buildroot-packages-directory"></a><a class="link" href="#buildroot-packages-directory">38.15.2. buildroot_packages directory</a></h4>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/">buildroot_packages/</a>.</p>
</div>
<div class="paragraph">
<p>Every directory inside it is a Buildroot package.</p>
</div>
<div class="paragraph">
<p>Those packages get automatically added to Buildroot&#8217;s <code>BR2_EXTERNAL</code>, so all you need to do is to turn them on during build, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_SAMPLE_PACKAGE=y'</pre>
</div>
</div>
<div class="paragraph">
<p>then test it out with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after '/sample_package.out'</pre>
</div>
</div>
<div class="paragraph">
<p>and you should see:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>hello sample_package</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/sample_package/sample_package.c">buildroot_packages/sample_package/sample_package.c</a></p>
</div>
<div class="paragraph">
<p>You can force a rebuild with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot --config 'BR2_PACKAGE_SAMPLE_PACKAGE=y' -- sample_package-reconfigure</pre>
</div>
</div>
<div class="paragraph">
<p>Buildroot packages are convenient, but in general, if a package if very important to you, but not really mergeable back to Buildroot, you might want to just use a custom build script for it, and point it to the Buildroot toolchain, and then use <code>BR2_ROOTFS_OVERLAY</code>, much like we do for <a href="#userland-setup">Userland setup</a>.</p>
</div>
<div class="paragraph">
<p>A custom build script can give you more flexibility: e.g. the package can be made work with other root filesystems more easily, have better <a href="#9p">9P</a> support, and rebuild faster as it evades some Buildroot boilerplate.</p>
</div>
<div class="sect4">
<h5 id="kernel-modules-buildroot-package"><a class="anchor" href="#kernel-modules-buildroot-package"></a><a class="link" href="#kernel-modules-buildroot-package">38.15.2.1. kernel_modules buildroot package</a></h5>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/kernel_modules/">buildroot_packages/kernel_modules/</a></p>
</div>
<div class="paragraph">
<p>An example of how to use kernel modules in Buildroot.</p>
</div>
<div class="paragraph">
<p>Usage:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-buildroot \
  --build-linux \
  --config 'BR2_PACKAGE_KERNEL_MODULES=y' \
  --no-overlay \
  -- \
  kernel_modules-reconfigure \
;</pre>
</div>
</div>
<div class="paragraph">
<p>Then test one of the modules with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --buildroot-linux --eval-after 'modprobe buildroot_hello'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/buildroot_packages/kernel_modules/buildroot_hello.c">buildroot_packages/kernel_modules/buildroot_hello.c</a></p>
</div>
<div class="paragraph">
<p>As you have just seen, this sets up everything so that <a href="#modprobe">modprobe</a> can correctly find the module.</p>
</div>
<div class="paragraph">
<p><code>./build-buildroot --build-linux</code> and <code>./run --buildroot-linux</code> are needed because the Buildroot kernel modules must use the Buildroot Linux kernel at build and run time, see also: <a href="#buildroot-vanilla-kernel">Buildroot vanilla kernel</a>.</p>
</div>
<div class="paragraph">
<p>The <code>--no-overlay</code> is required otherwise our <code>modules.order</code> generated by <code>./build-linux</code> and installed with <code>BR2_ROOTFS_OVERLAY</code> overwrites the Buildroot generated one.</p>
</div>
<div class="paragraph">
<p>Implementattion described at: <a href="https://stackoverflow.com/questions/40307328/how-to-add-a-linux-kernel-driver-module-as-a-buildroot-package/43874273#43874273" class="bare">https://stackoverflow.com/questions/40307328/how-to-add-a-linux-kernel-driver-module-as-a-buildroot-package/43874273#43874273</a></p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="patches-directory"><a class="anchor" href="#patches-directory"></a><a class="link" href="#patches-directory">38.15.3. patches directory</a></h4>
<div class="sect4">
<h5 id="patches-global-directory"><a class="anchor" href="#patches-global-directory"></a><a class="link" href="#patches-global-directory">38.15.3.1. patches/global directory</a></h5>
<div class="paragraph">
<p>Has the following structure:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>package-name/00001-do-something.patch</pre>
</div>
</div>
<div class="paragraph">
<p>The patches are then applied to the corresponding packages before build.</p>
</div>
<div class="paragraph">
<p>Uses <code>BR2_GLOBAL_PATCH_DIR</code>.</p>
</div>
</div>
<div class="sect4">
<h5 id="patches-manual-directory"><a class="anchor" href="#patches-manual-directory"></a><a class="link" href="#patches-manual-directory">38.15.3.2. patches/manual directory</a></h5>
<div class="paragraph">
<p>Patches in this directory are never applied automatically: it is up to users to manually apply them before usage following the instructions in this documentation.</p>
</div>
<div class="paragraph">
<p>These are typically patches that don&#8217;t contain fundamental functionality, so we don&#8217;t feel like forking the target repos.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="rootfs-overlay"><a class="anchor" href="#rootfs-overlay"></a><a class="link" href="#rootfs-overlay">38.15.4. rootfs_overlay</a></h4>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay">rootfs_overlay</a>.</p>
</div>
<div class="paragraph">
<p>We use this directory for:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>customized configuration files</p>
</li>
<li>
<p>userland module test scripts that don&#8217;t need to be compiled.</p>
<div class="paragraph">
<p>Contrast this with <a href="#userland-content">C examples</a> that need compilation.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>This directory is copied into the target filesystem by:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./copy-overlay
./build-buildroot</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/copy-overlay">copy-overlay</a></p>
</div>
<div class="paragraph">
<p>Build Buildroot is required for the same reason as described at: <a href="#your-first-kernel-module-hack">Section 2.2.2.2, &#8220;Your first kernel module hack&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>However, since the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay">rootfs_overlay</a> directory does not require compilation, unlike say <a href="#your-first-kernel-module-hack">kernel modules</a>, we also make it <a href="#9p">9P</a> available to the guest directly even without <code>./copy-overlay</code> at:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>ls /mnt/9p/rootfs_overlay</pre>
</div>
</div>
<div class="paragraph">
<p>This way you can just hack away the scripts and try them out immediately without any further operations.</p>
</div>
<div class="sect4">
<h5 id="out-rootfs-overlay-dir"><a class="anchor" href="#out-rootfs-overlay-dir"></a><a class="link" href="#out-rootfs-overlay-dir">38.15.4.1. <code>out_rootfs_overlay_dir</code></a></h5>
<div class="paragraph">
<p>This path can be found with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar out_rootfs_overlay_dir</pre>
</div>
</div>
<div class="paragraph">
<p>This output directory contains all the files that LKMC will put inside the final image, including for example:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#userland-content">Userland content</a> that needs to be compiled</p>
</li>
<li>
<p><a href="#rootfs-overlay">rootfs_overlay</a> content that gets put inside the image as is</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>LKMC first collects all the files that it will dump into the guest there, and then in the very last step dumps everything into the final image.</p>
</div>
<div class="paragraph">
<p>In Buildroot, this is done by pointing <code>BR2_ROOTFS_OVERLAY</code> to that directory, which is documented at: <a href="https://buildroot.org/downloads/manual/manual.html#rootfs-custom" class="bare">https://buildroot.org/downloads/manual/manual.html#rootfs-custom</a></p>
</div>
<div class="paragraph">
<p>This does not include native image modification mechanisms such as <a href="#buildroot-packages-directory">Buildroot packages</a>, which we let Buildroot itself manage.</p>
</div>
<div class="sect5">
<h6 id="disk-image-2"><a class="anchor" href="#disk-image-2"></a><a class="link" href="#disk-image-2">38.15.4.1.1. <code>disk_image_2</code></a></h6>
<div class="paragraph">
<p>A squashfs of <a href="#out-rootfs-overlay-dir"><code>out_rootfs_overlay_dir</code></a> that gets passed as the second argument.</p>
</div>
<div class="paragraph">
<p>Especially useful with <a href="#gem5">gem5</a> as a way to <a href="#gem5-restore-new-script">gem5 checkpoint restore and run a different script</a> via <a href="#secondary-disk">Secondary disk</a> since setting up <a href="#gem5-9p">gem5 9P</a> is slightly laborious.</p>
</div>
</div>
</div>
</div>
<div class="sect3">
<h4 id="lkmc-c"><a class="anchor" href="#lkmc-c"></a><a class="link" href="#lkmc-c">38.15.5. lkmc.c</a></h4>
<div class="paragraph">
<p>The files:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc.c">lkmc.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/lkmc.h">lkmc.h</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>contain common C function helpers that can be used both in userland and baremetal. Oh, the infinite <a href="#about-the-baremetal-setup">joys of Newlib</a>.</p>
</div>
<div class="paragraph">
<p>Those files also contain arch specific helpers under ifdefs like:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>#if defined(__aarch64__)</pre>
</div>
</div>
<div class="paragraph">
<p>We try to keep as much as possible in those files. It bloats builds a little, but just makes everything simpler to understand.</p>
</div>
<div class="paragraph">
<p>Link with lkmc.o is enabled with the <a href="#path-properties">path_properties.py</a></p>
</div>
<div class="literalblock">
<div class="content">
<pre>'extra_objs_lkmc_common': False,</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="lkmc-home"><a class="anchor" href="#lkmc-home"></a><a class="link" href="#lkmc-home">38.15.6. lkmc_home</a></h4>
<div class="paragraph">
<p><code>lkmc_home</code> refers to the target base directory in which we put all our custom built stuff, such as <a href="#userland-setup">userland executables</a> and <a href="#your-first-kernel-module-hack">kernel modules</a>.</p>
</div>
<div class="paragraph">
<p>The current value can be found with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getvar guest_lkmc_home</pre>
</div>
</div>
<div class="paragraph">
<p>In the past, we used to dump everything into the root filesystem, but as the userland structure got more complex with subfolders, we decided that the risk of conflicting with important root files was becoming too great.</p>
</div>
<div class="paragraph">
<p>To save you from typing that path every time, we have made our most common commands <code>cd</code> into that directory by default for you, e.g.:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>interactive shells <code>cd</code> there through <a href="#busybox-shell-initrc-files">BusyBox shell initrc files</a></p>
</li>
<li>
<p><code>--eval</code> and <code>--eval-after</code> through <a href="#replace-init">Replace init</a> and <a href="#init-busybox">Run command at the end of BusyBox init</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Whenever a relative path is used inside a guest sample command, e.g. <code>insmod hello.ko</code> or <code>./hello.out</code>, it means that the path lives in <code>lkmc_home</code> unless stated otherwise.</p>
</div>
</div>
<div class="sect3">
<h4 id="path-properties"><a class="anchor" href="#path-properties"></a><a class="link" href="#path-properties">38.15.7. path_properties.py</a></h4>
<div class="paragraph">
<p>In order to build and run each userland and <a href="#baremetal-setup">baremetal</a> example properly, we need per-file metadata such as compiler flags and required number of cores.</p>
</div>
<div class="paragraph">
<p>This data is stored is stored in <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/path_properties.py">path_properties.py</a> at <code>path_properties_tuples</code>.</p>
</div>
<div class="paragraph">
<p>Maybe we should embed it magically into source files directories to make it easier to see? But one big Python dict was easier to implement so we started like this. And it allows factoring chunks out easily.</p>
</div>
<div class="paragraph">
<p>The format is as follows:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>'path_component': (
    {'property': value},
    {
        'child_path_component':
        {
            {'child_property': },
            {}
        }
    }
)</pre>
</div>
</div>
<div class="paragraph">
<p>and as a shortcut, paths that don&#8217;t have any children can be written directly as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>'path_component': {'property': value}</pre>
</div>
</div>
<div class="paragraph">
<p>Properties of parent directories apply to all children.</p>
</div>
<div class="paragraph">
<p>Lists coming from parent directories are extended instead of overwritten by children, this is especially useful for C compiler flags.</p>
</div>
<div class="paragraph">
<p>To quickly determine which properties a path has, you can use <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/getprops">getprops</a>, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./getprops userland/c/hello.c</pre>
</div>
</div>
<div class="paragraph">
<p>which outputs values such as:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>allowed_archs=None
allowed_emulators=None
arm_aarch32=False
arm_sve=False
baremetal=True</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="rand-check-out"><a class="anchor" href="#rand-check-out"></a><a class="link" href="#rand-check-out">38.15.8. rand_check.out</a></h4>
<div class="paragraph">
<p>Print out several parameters that normally change randomly from boot to boot:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --eval-after './linux/rand_check.out;./linux/poweroff.out'</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/linux/rand_check.c">userland/linux/rand_check.c</a></p>
</div>
<div class="paragraph">
<p>This can be used to check the determinism of:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#norandmaps">norandmaps</a></p>
</li>
<li>
<p><a href="#qemu-record-and-replay">QEMU record and replay</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="test-this-repo"><a class="anchor" href="#test-this-repo"></a><a class="link" href="#test-this-repo">38.16. Test this repo</a></h3>
<div class="sect3">
<h4 id="automated-tests"><a class="anchor" href="#automated-tests"></a><a class="link" href="#automated-tests">38.16.1. Automated tests</a></h4>
<div class="paragraph">
<p>Run almost all tests:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build-test --all-archs --all-emulators --size 3 &amp;&amp; \
./test --size 3
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>should output 0.</p>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/build-test">build-test</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test">test</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test">test</a> script runs several different types of tests, which can also be run separately as explained at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-boot">test-boot</a></p>
</li>
<li>
<p><a href="#test-userland-in-full-system">Test userland in full system</a></p>
</li>
<li>
<p><a href="#user-mode-tests">User mode tests</a></p>
</li>
<li>
<p><a href="#baremetal-tests">Baremetal tests</a></p>
</li>
<li>
<p><a href="#gdb-tests">GDB tests</a></p>
</li>
<li>
<p><a href="#gem5-unit-tests">gem5 unit tests</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test">test</a> does not all possible tests, because there are too many possible variations and that would take forever. The rationale is the same as for <code>./build all</code> and is explained in <code>./build --help</code>.</p>
</div>
<div class="sect4">
<h5 id="test-arch-and-emulator-selection"><a class="anchor" href="#test-arch-and-emulator-selection"></a><a class="link" href="#test-arch-and-emulator-selection">38.16.1.1. Test arch and emulator selection</a></h5>
<div class="paragraph">
<p>You can select multiple archs and emulators of interest, as for an other command, with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables \
  --arch x86_64 \
  --arch aarch64 \
  --emulator gem5 \
  --emulator qemu \
;</pre>
</div>
</div>
<div class="paragraph">
<p>You can also test all supported archs and emulators with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables \
  --all-archs \
  --all-emulators \
;</pre>
</div>
</div>
<div class="paragraph">
<p>This command would run the test four times, using <code>x86_64</code> and <code>aarch64</code> with both gem5 and QEMU.</p>
</div>
<div class="paragraph">
<p>Without those flags, it defaults to just running the default arch and emulator once: <code>x86_64</code> and <code>qemu</code>.</p>
</div>
</div>
<div class="sect4">
<h5 id="quit-on-fail"><a class="anchor" href="#quit-on-fail"></a><a class="link" href="#quit-on-fail">38.16.1.2. Quit on fail</a></h5>
<div class="paragraph">
<p>By default, continue running even after the first failure happens, and they show a summary at the end.</p>
</div>
<div class="paragraph">
<p>You can make them exit immediately with the <code>--no-quit-on-fail</code> option, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-executables --quit-on-fail</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="test-userland-in-full-system"><a class="anchor" href="#test-userland-in-full-system"></a><a class="link" href="#test-userland-in-full-system">38.16.1.3. Test userland in full system</a></h5>
<div class="paragraph">
<p>TODO: we really need a mechanism to automatically generate the test list automatically e.g. based on <a href="#path-properties">path_properties.py</a>, currently there are many tests missing, and we have to add everything manually which is very annoying.</p>
</div>
<div class="paragraph">
<p>We could just generate it on the fly on the host, and forward it to guest through CLI arguments.</p>
</div>
<div class="paragraph">
<p>Run all userland tests from inside full system simulation (i.e. not <a href="#user-mode-simulation">User mode simulation</a>):</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-userland-full-system</pre>
</div>
</div>
<div class="paragraph">
<p>This includes, in particular, userland programs that test the kernel modules, which cannot be tested in user mode simulation.</p>
</div>
<div class="paragraph">
<p>Basically just boots and runs: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/test_all.sh">rootfs_overlay/lkmc/test_all.sh</a></p>
</div>
<div class="paragraph">
<p>Failure is detected by looking for the <a href="#magic-failure-string">Magic failure string</a></p>
</div>
<div class="paragraph">
<p>Most userland programs that don&#8217;t rely on kernel modules can also be tested in user mode simulation as explained at: <a href="#user-mode-tests">Section 11.2, &#8220;User mode tests&#8221;</a>.</p>
</div>
</div>
<div class="sect4">
<h5 id="gdb-tests"><a class="anchor" href="#gdb-tests"></a><a class="link" href="#gdb-tests">38.16.1.4. GDB tests</a></h5>
<div class="paragraph">
<p>We have some <a href="https://github.com/pexpect/pexpect">pexpect</a> automated tests for GDB for both userland and baremetal programs!</p>
</div>
<div class="paragraph">
<p>Run the userland tests:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --all-archs test-gdb &amp;&amp; \
./test-gdb --all-archs --all-emulators</pre>
</div>
</div>
<div class="paragraph">
<p>Run the baremetal tests instead:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./test-gdb --all-archs --all-emulators --mode baremetal</pre>
</div>
</div>
<div class="paragraph">
<p>Sources:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-gdb">test-gdb</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/gdb_tests/">userland/gdb_tests/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/arm/gdb_tests/">userland/arch/arm/gdb_tests/</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/arch/aarch64/gdb_tests/">userland/arch/aarch64/gdb_tests/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>If a test fails, re-run the test commands manually and use <code>--verbose</code> to understand what happened:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch arm --background --baremetal baremetal/c/add.c --gdb-wait &amp;
./run-gdb --arch arm --baremetal baremetal/c/add.c --verbose -- main</pre>
</div>
</div>
<div class="paragraph">
<p>and possibly repeat the GDB steps manually with the usual:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb --arch arm --baremetal baremetal/c/add.c --no-continue --verbose</pre>
</div>
</div>
<div class="paragraph">
<p>To debug GDB problems on gem5, you might want to enable the following <a href="#gem5-tracing">tracing</a> options:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run \
  --arch arm \
  --baremetal baremetal/c/add.c \
  --gdb-wait \
  --trace GDBRecv,GDBSend \
  --trace-stdout \
;</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="magic-failure-string"><a class="anchor" href="#magic-failure-string"></a><a class="link" href="#magic-failure-string">38.16.1.5. Magic failure string</a></h5>
<div class="paragraph">
<p>We do not know of any way to set the emulator exit status in QEMU arm full system.</p>
</div>
<div class="paragraph">
<p>For other arch / emulator combinations, we know how to do it:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>aarch64: aarch64 semihosting supports exit status</p>
</li>
<li>
<p>gem5: <a href="#m5-fail">m5 fail</a> works on all archs</p>
</li>
<li>
<p>user mode: QEMU forwards exit status, for gem5 we do some log parsing as described at: <a href="#gem5-syscall-emulation-exit-status">Section 11.7.2, &#8220;gem5 syscall emulation exit status&#8221;</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Since we can&#8217;t do it for QEMU arm, the only reliable solution is to just parse the guest serial output for a magic failure string to check if tests failed.</p>
</div>
<div class="paragraph">
<p>Our run scripts parse the serial output looking for a line line containing only exactly the magic regular expression:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>lkmc_exit_status_(\d+)</pre>
</div>
</div>
<div class="paragraph">
<p>and then exit with the given regular expression, e.g.:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch aarch64 baremetal/return2.c
echo $?</pre>
</div>
</div>
<div class="paragraph">
<p>should output:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>2</pre>
</div>
</div>
<div class="paragraph">
<p>This magic output string is notably generated by:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/rootfs_overlay/lkmc/test_fail.sh">rootfs_overlay/lkmc/test_fail.sh</a>, which is used by <a href="#test-userland-in-full-system">Test userland in full system</a></p>
</li>
<li>
<p>the <code>exit()</code> baremetal function when <code>status != 1</code>.</p>
<div class="paragraph">
<p>Unfortunately the only way we found to set this up was with <code>on_exit</code>: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59" class="bare">https://github.com/cirosantilli/linux-kernel-module-cheat/issues/59</a>.</p>
</div>
<div class="paragraph">
<p>Trying to patch <code>_exit</code> directly fails since at that point some de-initialization has already happened which prevents the print.</p>
</div>
<div class="paragraph">
<p>So setup this <code>on_exit</code> automatically from all our <a href="#baremetal-bootloaders">Baremetal bootloaders</a>, so it just works automatically for the examples that use the bootloaders: <a href="https://stackoverflow.com/questions/44097610/pass-parameter-to-atexit/49659697#49659697" class="bare">https://stackoverflow.com/questions/44097610/pass-parameter-to-atexit/49659697#49659697</a></p>
</div>
<div class="paragraph">
<p>The following examples end up testing that our setup is working:</p>
</div>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/assert_fail.c">userland/c/assert_fail.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return0.c">userland/c/return0.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return1.c">userland/c/return1.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/return2.c">userland/c/return2.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit0.c">userland/c/exit0.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit1.c">userland/c/exit1.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/c/exit2.c">userland/c/exit2.c</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/userland/posix/kill.c">userland/posix/kill.c</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Beware that on Linux kernel simulations, you cannot even echo that string from userland, since userland stdout shows up on the serial.</p>
</div>
</div>
</div>
<div class="sect3">
<h4 id="non-automated-tests"><a class="anchor" href="#non-automated-tests"></a><a class="link" href="#non-automated-tests">38.16.2. Non-automated tests</a></h4>
<div class="sect4">
<h5 id="test-gdb-linux-kernel"><a class="anchor" href="#test-gdb-linux-kernel"></a><a class="link" href="#test-gdb-linux-kernel">38.16.2.1. Test GDB Linux kernel</a></h5>
<div class="paragraph">
<p>For the Linux kernel, do the following manual tests for now.</p>
</div>
<div class="paragraph">
<p>Shell 1:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --gdb-wait</pre>
</div>
</div>
<div class="paragraph">
<p>Shell 2:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run-gdb start_kernel</pre>
</div>
</div>
<div class="paragraph">
<p>Should break GDB at <code>start_kernel</code>.</p>
</div>
<div class="paragraph">
<p>Then proceed to do the following tests:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>./count.sh</code> and <code>break __x64_sys_write</code></p>
</li>
<li>
<p><code>insmod timer.ko</code> and <code>break lkmc_timer_callback</code></p>
</li>
</ul>
</div>
</div>
<div class="sect4">
<h5 id="test-the-internet"><a class="anchor" href="#test-the-internet"></a><a class="link" href="#test-the-internet">38.16.2.2. Test the Internet</a></h5>
<div class="paragraph">
<p>You should also test that the Internet works:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./run --arch x86_64 --kernel-cli '- lkmc_eval="ifup -a;wget -S google.com;poweroff;"'</pre>
</div>
</div>
</div>
<div class="sect4">
<h5 id="cli-script-tests"><a class="anchor" href="#cli-script-tests"></a><a class="link" href="#cli-script-tests">38.16.2.3. CLI script tests</a></h5>
<div class="paragraph">
<p><code>build-userland</code> and <code>test-executables</code> have a wide variety of target selection modes, and it was hard to keep them all working without some tests:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-build-userland">test-build-userland</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/test-test-executables">test-test-executables</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect2">
<h3 id="bisection"><a class="anchor" href="#bisection"></a><a class="link" href="#bisection">38.17. Bisection</a></h3>
<div class="paragraph">
<p>When updating the Linux kernel, QEMU and gem5, things sometimes break.</p>
</div>
<div class="paragraph">
<p>However, for many types of crashes, it is trivial to bisect down to the offending commit, in particular because we can make QEMU and gem5 exit with status 1 on kernel panic as mentioned at: <a href="#exit-emulator-on-panic">Section 17.6.1.3, &#8220;Exit emulator on panic&#8221;</a>.</p>
</div>
<div class="paragraph">
<p>For example, when updating from QEMU <code>v2.12.0</code> to <code>v3.0.0-rc3</code>, the Linux kernel boot started to panic for <code>arm</code>.</p>
</div>
<div class="paragraph">
<p>We then bisected it as explained at: <a href="https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593" class="bare">https://stackoverflow.com/questions/4713088/how-to-use-git-bisect/22592593#22592593</a> with the <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/bisect-qemu-linux-boot">bisect-qemu-linux-boot</a> script:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>root_dir="$(pwd)"
cd "$(./getvar qemu_source_dir)"
git bisect start

# Check that our test script fails on v3.0.0-rc3 as expected, and mark it as bad.
"${root_dir}/bisect-qemu-linux-boot"
# Should output 1.
echo #?
git bisect bad

# Same for the good end.
git checkout v2.12.0
"${root_dir}/bisect-qemu-linux-boot"
# Should output 0.
echo #?
git bisect good

# This leaves us at the offending commit.
git bisect run "${root_dir}/bisect-qemu-linux-boot"

# Clean up after the bisection.
git bisect reset
git submodule update
"${root_dir}/build-qemu" --clean --qemu-build-id bisect</pre>
</div>
</div>
<div class="paragraph">
<p>Other bisection helpers include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/bisect-linux-boot-gem5">bisect-linux-boot-gem5</a></p>
</li>
<li>
<p><a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/bisect-gem5-linux-boot">bisect-gem5-linux-boot</a></p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="update-a-forked-submodule"><a class="anchor" href="#update-a-forked-submodule"></a><a class="link" href="#update-a-forked-submodule">38.18. Update a forked submodule</a></h3>
<div class="paragraph">
<p>This is a template update procedure for submodules for which we have some patches on on top of mainline.</p>
</div>
<div class="paragraph">
<p>This example is based on the Linux kernel, for which we used to have patches, but have since moved to mainline:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># Last point before out patches.
last_mainline_revision=v4.15
next_mainline_revision=v4.16
cd "$(./getvar linux_source_dir)"

# Create a branch before the rebase in case things go wrong.
git checkout -b "lkmc-${last_mainline_revision}"
git remote set-url origin git@github.com:cirosantilli/linux.git
git push
git checkout master

git fetch up
git rebase --onto "$next_mainline_revision" "$last_mainline_revision"

# And update the README to show off.
git commit -m "linux: update to ${next_mainline_revision}"</pre>
</div>
</div>
</div>
<div class="sect2">
<h3 id="release"><a class="anchor" href="#release"></a><a class="link" href="#release">38.19. Release</a></h3>
<div class="sect3">
<h4 id="release-procedure"><a class="anchor" href="#release-procedure"></a><a class="link" href="#release-procedure">38.19.1. Release procedure</a></h4>
<div class="paragraph">
<p>Ensure that the <a href="#automated-tests">Automated tests</a> are passing on a clean build:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>mv out out.bak
./build-test --size 3 &amp;&amp; ./test --size 3</pre>
</div>
</div>
<div class="paragraph">
<p>The <code>./build-test</code> command builds a superset of what will be downloaded which also tests other things we would like to be working on the release. For the minimal build to generate the files to be uploaded, see: <a href="#release-zip">Section 38.19.2, &#8220;release-zip&#8221;</a></p>
</div>
<div class="paragraph">
<p>The clean build is necessary as it generates clean images since <a href="#remove-buildroot-packages">it is not possible to remove Buildroot packages</a></p>
</div>
<div class="paragraph">
<p>Run all tests in <a href="#non-automated-tests">Non-automated tests</a> just QEMU x86_64 and QEMU aarch64.</p>
</div>
<div class="paragraph">
<p>TODO: not working currently, so skipped: Ensure that the <a href="#benchmark-this-repo">benchmarks</a> look fine:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./bench-all -A</pre>
</div>
</div>
<div class="paragraph">
<p>Create a release candidate and upload it:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git tag -a -m '' v3.0-rc1
git push --follow-tags
./release-zip --all-archs
# export LKMC_GITHUB_TOKEN=&lt;your-token&gt;
./release-upload</pre>
</div>
</div>
<div class="paragraph">
<p>Now let&#8217;s do an out-of-box testing for the release candidate:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd ..
git clone https://github.com/cirosantilli/linux-kernel-module-cheat linux-kernel-module-cheat-release
cd linux-kernel-module-cheat-release</pre>
</div>
</div>
<div class="paragraph">
<p>Test <a href="#prebuilt">Prebuilt setup</a>.</p>
</div>
<div class="paragraph">
<p>Clean up, and re-start from scratch:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>cd ..
rm -rf linux-kernel-module-cheat-release
git clone https://github.com/cirosantilli/linux-kernel-module-cheat linux-kernel-module-cheat-release
cd linux-kernel-module-cheat-release</pre>
</div>
</div>
<div class="paragraph">
<p>Go through all the other <a href="#getting-started">Getting started</a> sections in order.</p>
</div>
<div class="paragraph">
<p>Once everything looks fine, publish the release with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>git tag -a v3.0
# Describe the release int the tag message.
git push --follow-tags
./release-zip --all-archs
# export LKMC_GITHUB_TOKEN=&lt;your-token&gt;
./release-upload</pre>
</div>
</div>
</div>
<div class="sect3">
<h4 id="release-zip"><a class="anchor" href="#release-zip"></a><a class="link" href="#release-zip">38.19.2. release-zip</a></h4>
<div class="paragraph">
<p>Create a zip containing all files required for <a href="#prebuilt">Prebuilt setup</a>:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>./build --all-archs release &amp;&amp; ./release-zip --all-archs</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/release-zip">release-zip</a></p>
</div>
<div class="paragraph">
<p>This generates a zip file:</p>
</div>
<div class="literalblock">
<div class="content">
<pre>echo "$(./getvar release_zip_file)"</pre>
</div>
</div>
<div class="paragraph">
<p>which you can then upload somewhere.</p>
</div>
</div>
<div class="sect3">
<h4 id="release-upload"><a class="anchor" href="#release-upload"></a><a class="link" href="#release-upload">38.19.3. release-upload</a></h4>
<div class="paragraph">
<p>After:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>running <a href="#release-zip">release-zip</a></p>
</li>
<li>
<p>creating and pushing a tag to GitHub</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>you can upload the release to GitHub automatically with:</p>
</div>
<div class="literalblock">
<div class="content">
<pre># export LKMC_GITHUB_TOKEN=&lt;your-token&gt;
./release-upload</pre>
</div>
</div>
<div class="paragraph">
<p>Source: <a href="https://github.com/cirosantilli/linux-kernel-module-cheat/blob/master/release-upload">release-upload</a></p>
</div>
<div class="paragraph">
<p>The HEAD of the local repository must be on top of a tag that has been pushed for this to work.</p>
</div>
<div class="paragraph">
<p>Create <code>LKMC_GITHUB_TOKEN</code> under: <a href="https://github.com/settings/tokens/new" class="bare">https://github.com/settings/tokens/new</a> and save it to your <code>.bashrc</code>.</p>
</div>
<div class="paragraph">
<p>The implementation of this script is described at:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://stackoverflow.com/questions/5207269/how-to-release-a-build-artifact-asset-on-github-with-a-script/52354732#52354732" class="bare">https://stackoverflow.com/questions/5207269/how-to-release-a-build-artifact-asset-on-github-with-a-script/52354732#52354732</a></p>
</li>
<li>
<p><a href="https://stackoverflow.com/questions/38153418/can-someone-give-a-python-requests-example-of-uploading-a-release-asset-in-githu/52354681#52354681" class="bare">https://stackoverflow.com/questions/38153418/can-someone-give-a-python-requests-example-of-uploading-a-release-asset-in-githu/52354681#52354681</a></p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="design-rationale"><a class="anchor" href="#design-rationale"></a><a class="link" href="#design-rationale">38.20. Design rationale</a></h3>
<div class="sect3">
<h4 id="design-goals"><a class="anchor" href="#design-goals"></a><a class="link" href="#design-goals">38.20.1. Design goals</a></h4>
<div class="paragraph">
<p>This project was created to help me understand, modify and test low level system components by using system simulators.</p>
</div>
<div class="paragraph">
<p>System simulators are cool compared to real hardware because they are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>free</p>
</li>
<li>
<p>make experiments highly reproducible</p>
</li>
<li>
<p>give full visibility to the system: you can inspect any byte in memory, or the state of any hardware register. The laws of physics sometimes get in the way when doing that for real hardware.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The current components we focus the most on are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="#linux-kernel">Linux kernel</a> and Linux kernel modules</p>
</li>
<li>
<p>full systems emulators, currently <a href="#qemu-buildroot-setup">qemu</a> and <a href="#gem5-buildroot-setup">gem5</a></p>
</li>
<li>
<p><a href="#buildroot">Buildroot</a>. We use and therefore document, a large part of its feature set.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The following components are not covered, but they would also benefit from this setup, and it shouldn&#8217;t be hard to add them:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>C standard libraries</p>
</li>
<li>
<p>compilers. Project idea: add a new instruction to x86, then hack up GCC to actually use it, and make a C program that generates it.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>The design goals are to provide setups that are:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>highly automated: "just works"</p>
</li>
<li>
<p>thoroughly documented: you know what "just works" means</p>
</li>
<li>
<p>can be fully built from source: to give visibility and allow modifications</p>
</li>
<li>
<p>can also use <a href="#prebuilt">prebuilt binaries</a> as much as possible: in case you are lazy or unable to build from source</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>We aim to make a documentation that contains a very high runnable example / theory bullshit ratio.</p>
</div>
<div class="paragraph">
<p>Having at least one example per section is ideal, and it should be the very first thing in the section if possible.</p>
</div>
</div>
<div class="sect3">
<h4 id="setup-trade-offs"><a class="anchor" href="#setup-trade-offs"></a><a class="link" href="#setup-trade-offs">38.20.2. Setup trade-offs</a></h4>
<div class="paragraph">
<p>The trade-offs between the different <a href="#getting-started">setups</a> are basically a balance between:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>speed ans size: how long and how much disk space do the build and run take?</p>
</li>
<li>
<p>visibility: can you GDB step debug everything and read source code?</p>
</li>
<li>
<p>modifiability: can you modify the source code and rebuild a modified version?</p>
</li>
<li>
<p>portability: does it work on a Windows host? Could it ever?</p>
</li>
<li>
<p>accuracy: how accurate does the simulation represent real hardware?</p>
</li>
<li>
<p>compatibility: how likely is is that all the components will work well together: emulator, compiler, kernel, standard library, &#8230;&#8203;</p>
</li>
<li>
<p>guest software availability: how wide is your choice of easily installed guest software packages? See also: <a href="#linux-distro-choice">Section 38.20.4, &#8220;Linux distro choice&#8221;</a></p>
</li>
</ul>
</div>
</div>
<div class="sect3">
<h4 id="resource-tradeoff-guidelines"><a class="anchor" href="#resource-tradeoff-guidelines"></a><a class="link" href="#resource-tradeoff-guidelines">38.20.3. Resource tradeoff guidelines</a></h4>
<div class="paragraph">
<p>Choosing which features go into our default builds means making tradeoffs, here are our guidelines:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>keep the root filesystem as tiny as possible to make <a href="#prebuilt">Prebuilt setup</a> small: only add BusyBox to have a small interactive system.</p>
<div class="paragraph">
<p>It is easy to add new packages once you have the toolchain, and if you don&#8217;t there are infinitely many packages to cover and we can&#8217;t cover them all.</p>
</div>
</li>
<li>
<p>enable every feature possible on the toolchain (GCC, Binutils), because changes imply Buildroot rebuilds</p>
</li>
<li>
<p>runtime is sacred. Faster systems are:</p>
<div class="openblock">
<div class="content">
<div class="ulist">
<ul>
<li>
<p>easier to understand</p>
</li>
<li>
<p>run faster, which is specially for <a href="#gem5">gem5</a> which is slow</p>
</li>
</ul>
</div>
</div>
</div>
<div class="paragraph">
<p>Runtime basically just comes down to how we configure the Linux kernel, since in the root filesystem all that matters is <code>init=</code>, and that is easy to control.</p>
</div>
<div class="paragraph">
<p>One possibility we could play with is to build loadable modules instead of built-in modules to reduce runtime, but make it easier to get started with the modules.</p>
</div>
</li>
</ul>
</div>
<div class="paragraph">
<p>In order to learn how to measure some of those aspects, see: <a href="#benchmark-this-repo">Section 35, &#8220;Benchmark this repo&#8221;</a>.</p>
</div>
</div>
<div class="sect3">
<h4 id="linux-distro-choice"><a class="anchor" href="#linux-distro-choice"></a><a class="link" href="#linux-distro-choice">38.20.4. Linux distro choice</a></h4>
<div class="paragraph">
<p>We haven&#8217;t found the ultimate distro yet, here is a summary table of trade-offs that we care about: <a href="#table-lkmc-linux-distro-comparison">Table 8, &#8220;Comparison of Linux distros for usage in this repository&#8221;</a>.</p>
</div>
<table id="table-lkmc-linux-distro-comparison" class="tableblock frame-all grid-all stretch">
<caption class="title">Table 8. Comparison of Linux distros for usage in this repository</caption>
<colgroup>
<col style="width: 16.6666%;">
<col style="width: 16.6666%;">
<col style="width: 16.6666%;">
<col style="width: 16.6666%;">
<col style="width: 16.6666%;">
<col style="width: 16.667%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">Distro</th>
<th class="tableblock halign-left valign-top">Packages in single Git tree</th>
<th class="tableblock halign-left valign-top">Git tracked docs</th>
<th class="tableblock halign-left valign-top">Cross build without QEMU</th>
<th class="tableblock halign-left valign-top">Prebuilt downloads</th>
<th class="tableblock halign-left valign-top">Number of packages</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Buildroot 2018.05</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2k (1)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Ubuntu 18.04</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">50k (3)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Yocto 2.5 (8)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">?</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y (5)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">?</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y (6)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">400 (7)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">Alpine Linux 3.8.0</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">n (1)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">?</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">y</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">2000 (4)</p></td>
</tr>
</tbody>
</table>
<div class="ulist">
<ul>
<li>
<p>(1): Wiki&#8230;&#8203; <a href="https://wiki.alpinelinux.org/wiki/Main_Page" class="bare">https://wiki.alpinelinux.org/wiki/Main_Page</a></p>
</li>
<li>
<p>(2): <code>ls packages | wc</code></p>
</li>
<li>
<p>(3): <a href="https://askubuntu.com/questions/120630/how-many-packages-are-in-the-main-repository" class="bare">https://askubuntu.com/questions/120630/how-many-packages-are-in-the-main-repository</a></p>
</li>
<li>
<p>(4): <code>ls main community non-free | wc</code></p>
</li>
<li>
<p>(5): yes, but on a separate Git tree&#8230;&#8203; <a href="https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/" class="bare">https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/</a></p>
</li>
<li>
<p>(6): yes, but the initial Poky build / download still took 5 hours on <a href="#38mbps-internet">38Mbps internet</a>, and QEMU failed to boot at the end&#8230;&#8203; <a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=12953" class="bare">https://bugzilla.yoctoproject.org/show_bug.cgi?id=12953</a></p>
</li>
<li>
<p>(7): <code>ls recipes-* | wc</code></p>
</li>
<li>
<p>(8): Poky reference system: <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky" class="bare">http://git.yoctoproject.org/cgit/cgit.cgi/poky</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Other interesting possibilities that I haven&#8217;t evaluated well:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>NixOS <a href="https://nixos.org/" class="bare">https://nixos.org/</a> Seems to support full build from source well. Not much cross compilation information however.</p>
</li>
<li>
<p>Gentoo <a href="https://en.wikipedia.org/wiki/Gentoo_Linux" class="bare">https://en.wikipedia.org/wiki/Gentoo_Linux</a> Seems to support full build from source well.</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect2">
<h3 id="soft-topics"><a class="anchor" href="#soft-topics"></a><a class="link" href="#soft-topics">38.21. Soft topics</a></h3>
<div class="sect3">
<h4 id="fairy-tale"><a class="anchor" href="#fairy-tale"></a><a class="link" href="#fairy-tale">38.21.1. Fairy tale</a></h4>
<div class="quoteblock">
<blockquote>
<div class="paragraph">
<p>Once upon a time, there was a boy called Linus.</p>
</div>
<div class="paragraph">
<p>Linus made a super fun toy, and since he was not very humble, decided to call it Linux.</p>
</div>
<div class="paragraph">
<p>Linux was an awesome toy, but it had one big problem: it was very difficult to learn how to play with it!</p>
</div>
<div class="paragraph">
<p>As a result, only some weird kids who were very bored ended up playing with Linux, and everyone thought those kids were very cool, in their own weird way.</p>
</div>
<div class="paragraph">
<p>One day, a mysterious new kid called Ciro tried to play with Linux, and like many before him, got very frustrated, and gave up.</p>
</div>
<div class="paragraph">
<p>A few years later, Ciro had grown up a bit, and by chance came across a very cool toy made by the boy Petazzoni and his gang: it was called Buildroot.</p>
</div>
<div class="paragraph">
<p>Ciro noticed that if you used Buildroot together with Linux, and Linux suddenly became very fun to play with!</p>
</div>
<div class="paragraph">
<p>So Ciro decided to explain to as many kids as possible how to use Buildroot to play with Linux.</p>
</div>
<div class="paragraph">
<p>And so everyone was happy. Except some of the old weird kernel hackers who wanted to keep their mystique, but so be it.</p>
</div>
<div class="paragraph">
<p>THE END</p>
</div>
</blockquote>
</div>
</div>
</div>
<div class="sect2">
<h3 id="bibliography"><a class="anchor" href="#bibliography"></a><a class="link" href="#bibliography">38.22. Bibliography</a></h3>
<div class="paragraph">
<p>Runnable stuff:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://lwn.net/Kernel/LDD3/" class="bare">https://lwn.net/Kernel/LDD3/</a> the best book, but outdated. Updated source: <a href="https://github.com/martinezjavier/ldd3" class="bare">https://github.com/martinezjavier/ldd3</a> But examples non-minimal and take too much brain power to understand.</p>
</li>
<li>
<p><a href="https://github.com/satoru-takeuchi/elkdat" class="bare">https://github.com/satoru-takeuchi/elkdat</a> manual build process without Buildroot, very few and simple kernel modules. But it seem ktest + QEMU working, which is awesome. <code>./test</code> there patches ktest config dynamically based on CLI! Maybe we should just steal it since GPL licensed.</p>
</li>
<li>
<p><a href="https://github.com/tinyclub/linux-lab" class="bare">https://github.com/tinyclub/linux-lab</a> Buildroot based, no kernel modules?</p>
</li>
<li>
<p><a href="https://github.com/agelastic/eudyptula" class="bare">https://github.com/agelastic/eudyptula</a></p>
</li>
<li>
<p><a href="https://github.com/linux-kernel-labs" class="bare">https://github.com/linux-kernel-labs</a> Yocto based, source inside a kernel fork subdir: <a href="https://github.com/linux-kernel-labs/linux/tree/f08b9e4238dfc612a9d019e3705bd906930057fc/tools/labs" class="bare">https://github.com/linux-kernel-labs/linux/tree/f08b9e4238dfc612a9d019e3705bd906930057fc/tools/labs</a> which the author would like to upstream <a href="https://www.reddit.com/r/programming/comments/79w2q9/linux_device_driver_labs_the_linux_kernel/dp6of43/" class="bare">https://www.reddit.com/r/programming/comments/79w2q9/linux_device_driver_labs_the_linux_kernel/dp6of43/</a></p>
</li>
<li>
<p>Android AOSP: <a href="https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014" class="bare">https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014</a> AOSP is basically a uber bloated Buildroot (2 hours build vs 30 minutes), Android is Linux based, and QEMU is the emulator backend. These instructions might work for debugging the kernel: <a href="https://github.com/Fuzion24/AndroidKernelExploitationPlayground" class="bare">https://github.com/Fuzion24/AndroidKernelExploitationPlayground</a></p>
</li>
<li>
<p><a href="https://github.com/s-matyukevich/raspberry-pi-os" class="bare">https://github.com/s-matyukevich/raspberry-pi-os</a> Does both an OS from scratch, and annotates the corresponding kernel source code. For RPI3, no QEMU support: <a href="https://github.com/s-matyukevich/raspberry-pi-os/issues/8" class="bare">https://github.com/s-matyukevich/raspberry-pi-os/issues/8</a></p>
</li>
<li>
<p><a href="https://github.com/pw4ever/linux-kernel-hacking-helper" class="bare">https://github.com/pw4ever/linux-kernel-hacking-helper</a> as of bd9952127e7eda643cbb6cb4c51ad7b5b224f438, Bash, Arch Linux rootfs</p>
</li>
<li>
<p><a href="https://github.com/MichielDerhaeg/build-linux" class="bare">https://github.com/MichielDerhaeg/build-linux</a> untested. Manually builds musl and BusyBox, no Buildroot. Seems to use host packaged toolchain and tested on x86_64 only. Might contain a minimized kernel config.</p>
</li>
<li>
<p><a href="https://eli.thegreenplace.net" class="bare">https://eli.thegreenplace.net</a> and the accompanying code: <a href="https://github.com/eliben/code-for-blog" class="bare">https://github.com/eliben/code-for-blog</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Theory:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="http://cs241.cs.illinois.edu/coursebook/index.html" class="bare">http://cs241.cs.illinois.edu/coursebook/index.html</a> "CS 241: System Programming" from the University of Illinois at Urbana-Champaign. Has a PDF, Tex source at: <a href="https://github.com/illinois-cs241/coursebook" class="bare">https://github.com/illinois-cs241/coursebook</a> TODO any runnable code?</p>
</li>
<li>
<p><a href="https://github.com/0xAX/linux-insides" class="bare">https://github.com/0xAX/linux-insides</a> wait, how come they have 10x more starts as this repo? :-) Just kidding, awesome effort.</p>
</li>
<li>
<p><a href="http://nairobi-embedded.org" class="bare">http://nairobi-embedded.org</a> you will fall here a lot when you start popping the hard QEMU Google queries. They have covered everything we do here basically, but with a more manual approach, while this repo automates everything.</p>
<div class="paragraph">
<p>I couldn&#8217;t find the markup source code for the tutorials, and as a result when the domain went down in May 2018, you have to use <a href="http://web.archive.org/" class="bare">http://web.archive.org/</a> to see the pages&#8230;&#8203;</p>
</div>
</li>
<li>
<p><a href="https://balau82.wordpress.com" class="bare">https://balau82.wordpress.com</a> awesome low level resource</p>
</li>
<li>
<p><a href="https://rwmj.wordpress.com/" class="bare">https://rwmj.wordpress.com/</a> awesome red hatter</p>
</li>
<li>
<p><a href="https://lwn.net" class="bare">https://lwn.net</a></p>
</li>
<li>
<p><a href="http://www.makelinux.net" class="bare">http://www.makelinux.net</a></p>
</li>
<li>
<p><a href="https://notes.shichao.io/lkd/" class="bare">https://notes.shichao.io/lkd/</a></p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Awesome lists:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><a href="https://github.com/gurugio/lowlevelprogramming-university" class="bare">https://github.com/gurugio/lowlevelprogramming-university</a></p>
</li>
<li>
<p><a href="https://github.com/uhub/awesome-c" class="bare">https://github.com/uhub/awesome-c</a></p>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<script>
<!-- Google Analytics, AKA selling my soul to Google for some backlinks. -->
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-47867706-1', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>